summaryrefslogtreecommitdiff
path: root/arch/arm/kernel/smp_tlb.c
AgeCommit message (Collapse)Author
2019-06-19treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500Thomas Gleixner
Based on 2 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation # extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to replace the boilerplate/reference in 4122 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Enrico Weigelt <info@metux.net> Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org> Reviewed-by: Allison Randal <allison@lohutok.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190604081206.933168790@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-01-16ARM: 8613/1: Fix the uaccess crash on PB11MPCoreLinus Walleij
The following patch was sketched by Russell in response to my crashes on the PB11MPCore after the patch for software-based priviledged no access support for ARMv8.1. See this thread: http://marc.info/?l=linux-arm-kernel&m=144051749807214&w=2 I am unsure what is going on, I suspect everyone involved in the discussion is. I just want to repost this to get the discussion restarted, as I still have to apply this patch with every kernel iteration to get my PB11MPCore Realview running. Testing by Neil Armstrong on the Oxnas NAS has revealed that this bug exist also on that widely deployed hardware, so we are probably currently regressing all ARM11MPCore systems. Cc: Russell King <linux@armlinux.org.uk> Cc: Will Deacon <will.deacon@arm.com> Fixes: a5e090acbf54 ("ARM: software-based priviledged-no-access support") Tested-by: Neil Armstrong <narmstrong@baylibre.com> Signed-off-by: Linus Walleij <linus.walleij@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2016-07-02ARM: 8577/1: Fix Cortex-A15 798181 errata initializationMatija Glavinic Pecotic
Current errata initialization doesn't take properly revision and REVIDR into account. Depending on the core revision, revidr bits should not be taken into account. Errata misleadingly declares r3p3 to be error-free, but this is not the case. Include rp3p3 in errata initialization. Here are possible fixes defined in revidr register for r2 and r3 [1,2]: r0p0-r2p1: No fixes applied r2p2,r2p3: REVIDR[4]: 798181 Moving a virtual page that is being accessed by an active process can lead to unexpected behavior REVIDR[9]: Not defined r2p4,r3p0,r3p1,r3p2: REVIDR[4]: 798181 Moving a virtual page that is being accessed by an active process can lead to unexpected behavior REVIDR[9]: 798181 Moving a virtual page that is being accessed by an active process can lead to unexpected behavior - This is an update to a previously released ECO. r3p3: REVIDR[4]: Reserved REVIDR[9]: 798181 Moving a virtual page that is being accessed by an active process can lead to unexpected behavior - This is an update to a previously released ECO. And here is proposed handling from the same document: * In r3p2 and earlier versions with REVIDR[4]= 0,the full workaround is required. * In r3p2 and earlier versions with REVIDR[4]=1, REVIDR[9]=0, only the portion of the workaround up to the end of step 6 is required. * In r3p2 and earlier versions with REVIDR[4]=1, REVIDR[9]=1, no workaround is required. * In r3p3, if REVIDR[9]=0, only the portion of the workaround up to the end of step 6 is required. * In r3p3, if REVIDR[9]=1, no workaround is required. These imply following: REVIDR[9] set -> No WA REVIDR[4] set, REVIDR[9] cleared -> Partial WA Both cleared -> Full WA Where certain bits should and should not be taken into account depending on whether they are defined for the revision. Although not explicitly mentioned in the errata note, REVIDR[9] set, with REVIDR[4] cleared is valid combination which requires no WA. This is confirmed by ARM support and errata will be updated. [1] ARM CortexTM-A15 MPCore - NEON Product revision r3 Software Developers Errata Notice ARM-EPM-028093 v20.0 Released [2] ARM CortexTM-A15 MPCore - NEON Product Revision r2 Software Developers Errata Notice ARM-EPM-028090 v19.3 Released Signed-off-by: Matija Glavinic Pecotic <matija.glavinic-pecotic.ext@nokia.com> Reviewed-by: Alexander Sverdlin <alexander.sverdlin@nokia.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-07-24ARM: 8111/1: Enable erratum 798181 for Broadcom Brahma-B15Gregory Fong
Broadcom Brahma-B15 (r0p0..r0p2) is also affected by Cortex-A15 erratum 798181, so enable the workaround for Brahma-B15. Signed-off-by: Gregory Fong <gregory.0xf0@gmail.com> Acked-by: Marc Carino <marc.ceeeee@gmail.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Brian Norris <computersforpeace@gmail.com> Cc: Rob Herring <rob.herring@calxeda.com> Cc: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-10-29ARM: 7855/1: Add check for Cortex-A15 errata 798181 ECORob Herring
The work-around for A15 errata 798181 is not needed if appropriate ECO fixes have been applied to r3p2 and earlier core revisions. This can be checked by reading REVIDR register bits 4 and 9. If only bit 4 is set, then the IPI broadcast can be skipped. Signed-off-by: Rob Herring <rob.herring@calxeda.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-08-12ARM: tlb: don't perform inner-shareable invalidation for local BP opsWill Deacon
Now that the ASID allocator doesn't require inner-shareable maintenance, we can convert the local_bp_flush_all function to perform only non-shareable flushing, in a similar manner to the TLB invalidation routines. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2013-08-12ARM: tlb: don't perform inner-shareable invalidation for local TLB opsWill Deacon
Inner-shareable TLB invalidation is typically more expensive than local (non-shareable) invalidation, so performing the broadcasting for local_flush_tlb_* operations is a waste of cycles and needlessly clobbers entries in the TLBs of other CPUs. This patch introduces __flush_tlb_* versions for many of the TLB invalidation functions, which only respect inner-shareable variants of the invalidation instructions when presented with the TLB_V7_UIS_FULL flag. The local version is also inlined to prevent SMP_ON_UP kernels from missing flushes, where the __flush variant would be called with the UP flags. This gains us around 0.5% in hackbench scores for a dual-core A15, but I would expect this to improve as more cores (and clusters) are added to the equation. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Reported-by: Albin Tonnerre <Albin.Tonnerre@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2013-07-26ARM: 7789/1: Do not run dummy_flush_tlb_a15_erratum() on non-Cortex-A15Fabio Estevam
Commit 93dc688 (ARM: 7684/1: errata: Workaround for Cortex-A15 erratum 798181 (TLBI/DSB operations)) causes the following undefined instruction error on a mx53 (Cortex-A8): Internal error: Oops - undefined instruction: 0 [#1] SMP ARM CPU: 0 PID: 275 Comm: modprobe Not tainted 3.11.0-rc2-next-20130722-00009-g9b0f371 #881 task: df46cc00 ti: df48e000 task.ti: df48e000 PC is at check_and_switch_context+0x17c/0x4d0 LR is at check_and_switch_context+0xdc/0x4d0 This problem happens because check_and_switch_context() calls dummy_flush_tlb_a15_erratum() without checking if we are really running on a Cortex-A15 or not. To avoid this issue, only call dummy_flush_tlb_a15_erratum() inside check_and_switch_context() if erratum_a15_798181() returns true, which means that we are really running on a Cortex-A15. Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Reviewed-by: Roger Quadros <rogerq@ti.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-06-24ARM: 7769/1: Cortex-A15: fix erratum 798181 implementationMarc Zyngier
Looking into the active_asids array is not enough, as we also need to look into the reserved_asids array (they both represent processes that are currently running). Also, not holding the ASID allocator lock is racy, as another CPU could schedule that process and trigger a rollover, making the erratum workaround miss an IPI. Exposing this outside of context.c is a little ugly on the side, so let's define a new entry point that the erratum workaround can call to obtain the cpumask. Cc: <stable@vger.kernel.org> # 3.9 Acked-by: Will Deacon <will.deacon@arm.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-04-25ARM: 7703/1: Disable preemption in broadcast_tlb*_a15_erratum()Catalin Marinas
Commit 93dc688 (ARM: 7684/1: errata: Workaround for Cortex-A15 erratum 798181 (TLBI/DSB operations)) introduces calls to smp_processor_id() and smp_call_function_many() with preemption enabled. This patch disables preemption and also optimises the smp_processor_id() call in broadcast_tlb_mm_a15_erratum(). The broadcast_tlb_a15_erratum() function is changed to use smp_call_function() which disables preemption. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Reported-by: Geoff Levand <geoff@infradead.org> Reported-by: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-04-03ARM: 7684/1: errata: Workaround for Cortex-A15 erratum 798181 (TLBI/DSB ↵Catalin Marinas
operations) On Cortex-A15 (r0p0..r3p2) the TLBI/DSB are not adequately shooting down all use of the old entries. This patch implements the erratum workaround which consists of: 1. Dummy TLBIMVAIS and DSB on the CPU doing the TLBI operation. 2. Send IPI to the CPUs that are running the same mm (and ASID) as the one being invalidated (or all the online CPUs for global pages). 3. CPU receiving the IPI executes a DMB and CLREX (part of the exception return code already). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-03-03ARM: 7660/1: tlb: add branch predictor maintenance operationsWill Deacon
The ARM architecture requires explicit branch predictor maintenance when updating an instruction stream for a given virtual address. In reality, this isn't so much of a burden because the branch predictor is flushed during the cache maintenance required to make the new instructions visible to the I-side of the processor. However, there are still some cases where explicit flushing is required, so add a local_bp_flush_all operation to deal with this. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-03-28smp: introduce a generic on_each_cpu_mask() functionGilad Ben-Yossef
We have lots of infrastructure in place to partition multi-core systems such that we have a group of CPUs that are dedicated to specific task: cgroups, scheduler and interrupt affinity, and cpuisol= boot parameter. Still, kernel code will at times interrupt all CPUs in the system via IPIs for various needs. These IPIs are useful and cannot be avoided altogether, but in certain cases it is possible to interrupt only specific CPUs that have useful work to do and not the entire system. This patch set, inspired by discussions with Peter Zijlstra and Frederic Weisbecker when testing the nohz task patch set, is a first stab at trying to explore doing this by locating the places where such global IPI calls are being made and turning the global IPI into an IPI for a specific group of CPUs. The purpose of the patch set is to get feedback if this is the right way to go for dealing with this issue and indeed, if the issue is even worth dealing with at all. Based on the feedback from this patch set I plan to offer further patches that address similar issue in other code paths. This patch creates an on_each_cpu_mask() and on_each_cpu_cond() infrastructure API (the former derived from existing arch specific versions in Tile and Arm) and uses them to turn several global IPI invocation to per CPU group invocations. Core kernel: on_each_cpu_mask() calls a function on processors specified by cpumask, which may or may not include the local processor. You must not call this function with disabled interrupts or from a hardware interrupt handler or from a bottom half handler. arch/arm: Note that the generic version is a little different then the Arm one: 1. It has the mask as first parameter 2. It calls the function on the calling CPU with interrupts disabled, but this should be OK since the function is called on the other CPUs with interrupts disabled anyway. arch/tile: The API is the same as the tile private one, but the generic version also calls the function on the with interrupts disabled in UP case This is OK since the function is called on the other CPUs with interrupts disabled. Signed-off-by: Gilad Ben-Yossef <gilad@benyossef.com> Reviewed-by: Christoph Lameter <cl@linux.com> Acked-by: Chris Metcalf <cmetcalf@tilera.com> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Russell King <linux@arm.linux.org.uk> Cc: Pekka Enberg <penberg@kernel.org> Cc: Matt Mackall <mpm@selenic.com> Cc: Rik van Riel <riel@redhat.com> Cc: Andi Kleen <andi@firstfloor.org> Cc: Sasha Levin <levinsasha928@gmail.com> Cc: Mel Gorman <mel@csn.ul.ie> Cc: Alexander Viro <viro@zeniv.linux.org.uk> Cc: Avi Kivity <avi@redhat.com> Acked-by: Michal Nazarewicz <mina86@mina86.org> Cc: Kosaki Motohiro <kosaki.motohiro@gmail.com> Cc: Milton Miller <miltonm@bga.com> Cc: Russell King <linux@arm.linux.org.uk> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2010-12-20ARM: SMP: split out software TLB maintainence broadcastingRussell King
smp.c is becoming too large, so split out the TLB maintainence broadcasting into a separate smp_tlb.c file. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>