dmnt2: fix gdb register output

This also fixes ctrl-c break in gdbstub, and fixes crash on unknown monitor cmd.

.gitignore

@@ -91,7 +91,11 @@ dkms.conf
 **/out
 **/build
 **/build_nintendo_nx_arm64
+**/build_nintendo_nx_arm64_armv8a
 **/build_nintendo_nx_arm
+**/build_nintendo_nx_arm_armv8a
+**/build_nintendo_nx_arm_armv7a
+**/build_nintendo_nx_arm_armv4t
 **/build_nintendo_nx_x64
 **/build_nintendo_nx_x86
 

Makefile

@@ -112,6 +112,7 @@ dist-no-debug: all
 	mkdir -p atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/0100000000000042
 	mkdir -p atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/0100000000000420
 	mkdir -p atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/010000000000B240
+	mkdir -p atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/010000000000D609
 	mkdir -p atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/010000000000D623
 	cp stratosphere/boot2/boot2.nsp atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/0100000000000008/exefs.nsp
 	cp stratosphere/dmnt/dmnt.nsp atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/010000000000000D/exefs.nsp
@@ -125,6 +126,7 @@ dist-no-debug: all
 	cp stratosphere/pgl/pgl.nsp atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/0100000000000042/exefs.nsp
 	cp stratosphere/LogManager/LogManager.nsp atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/0100000000000420/exefs.nsp
 	cp stratosphere/htc/htc.nsp atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/010000000000B240/exefs.nsp
+	cp stratosphere/dmnt.gen2/dmnt.gen2.nsp atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/010000000000D609/exefs.nsp
 	cp stratosphere/TioServer/TioServer.nsp atmosphere-$(AMSVER)/stratosphere_romfs/atmosphere/contents/010000000000D623/exefs.nsp
 	@build_romfs atmosphere-$(AMSVER)/stratosphere_romfs atmosphere-$(AMSVER)/atmosphere/stratosphere.romfs
 	rm -r atmosphere-$(AMSVER)/stratosphere_romfs

docs/changelog.md

@@ -1,4 +1,59 @@
 # Changelog
+## 1.2.2
++ A number of fixes were made to Atmosphère's implementation of the new "sprofile" service added in 13.0.0.
+  + Nintendo is finally transmitting data over the internet to certain consoles, which has allowed for validating our service implementation.
+    + Unfortunately, there were several problems, and if your console began trying to use the new services atmosphere would show a fatal error with code 0xCAF6 (sprofile::ResultInvalidState()).
+  + With actual test data in hand, a test program was written and it was verified that our implementation can successfully import/access profile data now.
+    + Hopefully there are no more issues, and I sincerely apologize for anyone who got an 0xCAF6 fatal due to this.
++ A number of minor improvements were made to `mesosphère`, including:
+  + KThread::GetContextForSchedulerLoop was implemented in assembly (using static assertions to verify offset-of-context-in-struct is correct).
+    + This saves an unnecessary function call in the middle of the scheduler hot loop, replacing it with an addition instruction, which should improve microperformance.
+  + Mesosphere's hardware maintenance instructions were audited via a script and now directly match Nintendo's kernels.
+    + Notably, this inserts a missing instruction synchronization barrier when validating that slab heaps may be constructed.
+    + This missing ISB could cause an abort on certain (see: particularly sensitive) hardware on boot if the relevant codepath was speculatively executed (it normally only executes on game launch...)
+  + The SVC handlers for performing light IPC (normally unused) from 32-bit process were fixed in Mesosphere.
+  + A bug was fixed that would cause the register x27 to be overwritten with the contents of x26 when returning from a user exception handler.
+  + A bug was fixed that would cause the kernel to use the userland stack pointer instead of the kernel stack pointer while generating an error report for a kernel abort.
++ General system stability improvements to enhance the user's experience.
+## 1.2.1
++ Support was implemented for 13.1.0.
+  + `mesosphère` was updated to reflect the kernel behavioral changes made in 13.1.0.
+    + KScheduler now issues a data memory barrier when unlocking the scheduler lock and when early-returning due to top-thread-is-current during scheduling.
+  + `erpt` was  updated to reflect the latest official behaviors.
+    + The new service added in 13.0.0 ("sprofile") was revised, and the data formats it expects was changed.
+      + This still appears to be (possibly(?)) untestable due to data not being transmitted yet, but I have greater confidence things will go smoothly than I did when 1.1.0 released.
++ A number of improvements were made to `mesosphère`, including:
+  + A build target was created to build targeting the qemu `virt` board.
+    + This facilitates writing unit tests for the kernel (and other atmosphere components) and running them under PC.
+      + **Please Note**: Official system software will not work at all under this, and the Atmosphère project has zero interest in attempting to run official software of any kind. This is unit testing machinery, and explicitly not more than that.
+    + This should hopefully allow us to have greater confidence that all of atmosphere's components work the way they're theoretically supposed to in the future.
+    + **Please Note**: If you are a developer who is familiar with the Horizon operating system (or capable of becoming familiar), I would greatly appreciate help writing tests and improving the testing framework.
+      + Please contact `SciresM#0524` if you are capable and interested.
+        + Really, if you are actually a developer who would like to help me get this off the ground, I would deeply appreciate it.
+        + That said, if you are not a developer but want to be one, this probably isn't the best opportunity; I expect it to be highly technical.
+          + Consider the ReSwitched discord's #hack-n-all channel for your educational purposes.
+      + We are (at least for now) using [catch2](https://github.com/catchorg/Catch2) for unit tests.
+  + Almost all virtual calls in the kernel are now resolved statically.
+    + This eliminates substantial virtual call overhead, and should lead to improved kernel microperformance in pretty much every function.
+  + The remaining red black tree find operations which weren't using the optimized "find key" variant are now using the optimized version.
+  + Custom assembly was written to improve tick-to-timespan conversion.
+    + This works around gcc emitting suboptimal assembly at -Os (it emits good assembly at -O3, clang is fine at both -O3 and -Os).
+  + KThread and KSession structures were updated to optimize member layout, saving 0x10 bytes per KThread/KSession object.
+  + Rather than unnecessarily zero-ing all data in kernel objects only to overwrite members later, we now only initialize the members we need to in kernel object constructors.
+    + This is what Nintendo was doing already.
+  + A set of custom optimized atomic primitives were implemented and are used in place of std::atomic<>
+    + This works around a gcc bug which downgrades specified memory order to seq_cst, and introduces clrex in places where it is appropriate.
+    + This should strictly improve microperformance of many system calls.
+  + An compile-time toggleable extension was added to support 40-bit physical addresses in MapRange capabilities (using currently reserved bits).
+  + A number of minor bugs were fixed, including:
+    + Initial cache management now better reflects official behavior.
+      + This fixes an issue that caused certain hardware with cache sensitivity to produce cryptic kernel panics during boot.
+    + Incorrect logic when checking thread priority capabilities was fixed to reflect official behavior.
+    + The scheduler was updated to reflect latest official behavior, and a number of minor bugs involving clz/ctz were fixed.
+    + Accesses to the processes local region were fixed to properly use kernel linear region, not userland pointers.
+    + The cache SVCs exposed for 32-bit processes now better reflect official core mask request semantics.
+    + A bug was fixed that could cause a kernel panic if SvcArbitrateLock was called on a thread with exactly one reference in the middle of handling a user-mode exception.
++ General system stability improvements to enhance the user's experience.
 ## 1.2.0
 + `boot` was updated to reflect the latest official behavior for display/battery management.
   + This should fix any issues that might result from running older releases on the OLED model, if you're somehow in a position to do so.

emummc/.gitrepo

@@ -6,7 +6,7 @@
 [subrepo]
 	remote = https://github.com/m4xw/emuMMC
 	branch = develop
-	commit = f66087313546161a000ee196a788f0626caf80fa
-	parent = 38f9a76ba028995ed3274da3a45b0254f09d1f59
-	method = rebase
+	commit = a9d56959460fc794ce2cb6405402c25a3e89c47f
+	parent = ff719641396c635b735873fb2b020c910f768a04
+	method = merge
 	cmdver = 0.4.1

emummc/README.md

@@ -1,21 +1,21 @@
 # emuMMC
-*A SDMMC driver replacement for Nintendo's Filesystem Services, by **m4xw***
+*A SDMMC driver replacement for Nintendo's Filesystem Services, by **m4xw***  
 
 ### Supported Horizon Versions
-**1.0.0 - 13.0.0**
+**1.0.0 - 13.1.0**
 
 ## Features
-* Arbitrary SDMMC backend selection
+* Arbitrary SDMMC backend selection  
   **This allows loading eMMC from SD or even SD from eMMC**
-* On the fly hooking / patching, fully self-infesting
+* On the fly hooking / patching, fully self-infesting  
   **Only one payload required for all versions!**
-* File-based SDMMC backend support (from SD)
+* File-based SDMMC backend support (from SD)  
   **This allows loading eMMC images from hekate-backups (split or not)**
-* SDMMC device based sector offset (*currently eMMC only*)
+* SDMMC device based sector offset (*currently eMMC only*)  
   **Raw partition support for eMMC from SD with less performance overhead**
-* Full support for `/Nintendo` folder redirection to a arbitrary path
+* Full support for `/Nintendo` folder redirection to a arbitrary path  
   **No 8 char length restriction!**
-* exosphere based context configuration
+* exosphere based context configuration  
   **This includes full support for multiple emuMMC images**
 
 ## Compiling

emummc/source/FS/FS_offsets.c

@@ -57,6 +57,8 @@
 #include "offsets/1203_exfat.h"
 #include "offsets/1300.h"
 #include "offsets/1300_exfat.h"
+#include "offsets/1310.h"
+#include "offsets/1310_exfat.h"
 #include "../utils/fatal.h"
 
 #define GET_OFFSET_STRUCT_NAME(vers) g_offsets##vers
@@ -125,6 +127,8 @@ DEFINE_OFFSET_STRUCT(_1203);
 DEFINE_OFFSET_STRUCT(_1203_EXFAT);
 DEFINE_OFFSET_STRUCT(_1300);
 DEFINE_OFFSET_STRUCT(_1300_EXFAT);
+DEFINE_OFFSET_STRUCT(_1310);
+DEFINE_OFFSET_STRUCT(_1310_EXFAT);
 
 const fs_offsets_t *get_fs_offsets(enum FS_VER version) {
     switch (version) {
@@ -210,6 +214,10 @@ const fs_offsets_t *get_fs_offsets(enum FS_VER version) {
             return &(GET_OFFSET_STRUCT_NAME(_1300));
         case FS_VER_13_0_0_EXFAT:
             return &(GET_OFFSET_STRUCT_NAME(_1300_EXFAT));
+        case FS_VER_13_1_0:
+            return &(GET_OFFSET_STRUCT_NAME(_1310));
+        case FS_VER_13_1_0_EXFAT:
+            return &(GET_OFFSET_STRUCT_NAME(_1310_EXFAT));
         default:
             fatal_abort(Fatal_UnknownVersion);
     }

emummc/source/FS/FS_versions.h

@@ -83,6 +83,9 @@ enum FS_VER
     FS_VER_13_0_0,
     FS_VER_13_0_0_EXFAT,
 
+    FS_VER_13_1_0,
+    FS_VER_13_1_0_EXFAT,
+
     FS_VER_MAX,
 };
 

emummc/source/FS/offsets/1310.h

@@ -0,0 +1,59 @@
+/*
+ * Copyright (c) 2019 m4xw <m4x@m4xw.net>
+ * Copyright (c) 2019 Atmosphere-NX
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __FS_1310_H__
+#define __FS_1310_H__
+
+// Accessor vtable getters
+#define FS_OFFSET_1310_SDMMC_ACCESSOR_GC   0x158C20
+#define FS_OFFSET_1310_SDMMC_ACCESSOR_SD   0x15AA30
+#define FS_OFFSET_1310_SDMMC_ACCESSOR_NAND 0x159150
+
+// Hooks
+#define FS_OFFSET_1310_SDMMC_WRAPPER_READ  0x1545C0
+#define FS_OFFSET_1310_SDMMC_WRAPPER_WRITE 0x154680
+#define FS_OFFSET_1310_RTLD                0x688
+#define FS_OFFSET_1310_RTLD_DESTINATION    ((uintptr_t)(INT64_C(-0x3C)))
+
+#define FS_OFFSET_1310_CLKRST_SET_MIN_V_CLK_RATE 0x1537C0
+
+// Misc funcs
+#define FS_OFFSET_1310_LOCK_MUTEX          0x29690
+#define FS_OFFSET_1310_UNLOCK_MUTEX        0x296E0
+
+#define FS_OFFSET_1310_SDMMC_WRAPPER_CONTROLLER_OPEN  0x1544A0
+#define FS_OFFSET_1310_SDMMC_WRAPPER_CONTROLLER_CLOSE 0x154530
+
+// Misc Data
+#define FS_OFFSET_1310_SD_MUTEX            0xE133E8
+#define FS_OFFSET_1310_NAND_MUTEX          0xE0E768
+#define FS_OFFSET_1310_ACTIVE_PARTITION    0xE0E7A8
+#define FS_OFFSET_1310_SDMMC_DAS_HANDLE    0xDF6E18
+
+// NOPs
+#define FS_OFFSET_1310_SD_DAS_INIT         0x27744
+
+// Nintendo Paths
+#define FS_OFFSET_1310_NINTENDO_PATHS \
+{ \
+	{.opcode_reg = 3, .adrp_offset = 0x0006EBE0, .add_rel_offset = 0x00000004}, \
+	{.opcode_reg = 3, .adrp_offset = 0x0007BEEC, .add_rel_offset = 0x00000004}, \
+	{.opcode_reg = 4, .adrp_offset = 0x00082294, .add_rel_offset = 0x00000004}, \
+	{.opcode_reg = 4, .adrp_offset = 0x0009422C, .add_rel_offset = 0x00000004}, \
+	{.opcode_reg = 0, .adrp_offset = 0, .add_rel_offset = 0}, \
+}
+
+#endif // __FS_1310_H__

emummc/source/FS/offsets/1310_exfat.h

@@ -0,0 +1,59 @@
+/*
+ * Copyright (c) 2019 m4xw <m4x@m4xw.net>
+ * Copyright (c) 2019 Atmosphere-NX
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __FS_1310_EXFAT_H__
+#define __FS_1310_EXFAT_H__
+
+// Accessor vtable getters
+#define FS_OFFSET_1310_EXFAT_SDMMC_ACCESSOR_GC   0x158C20
+#define FS_OFFSET_1310_EXFAT_SDMMC_ACCESSOR_SD   0x15AA30
+#define FS_OFFSET_1310_EXFAT_SDMMC_ACCESSOR_NAND 0x159150
+
+// Hooks
+#define FS_OFFSET_1310_EXFAT_SDMMC_WRAPPER_READ  0x1545C0
+#define FS_OFFSET_1310_EXFAT_SDMMC_WRAPPER_WRITE 0x154680
+#define FS_OFFSET_1310_EXFAT_RTLD                0x688
+#define FS_OFFSET_1310_EXFAT_RTLD_DESTINATION    ((uintptr_t)(INT64_C(-0x3C)))
+
+#define FS_OFFSET_1310_EXFAT_CLKRST_SET_MIN_V_CLK_RATE 0x1537C0
+
+// Misc funcs
+#define FS_OFFSET_1310_EXFAT_LOCK_MUTEX          0x29690
+#define FS_OFFSET_1310_EXFAT_UNLOCK_MUTEX        0x296E0
+
+#define FS_OFFSET_1310_EXFAT_SDMMC_WRAPPER_CONTROLLER_OPEN  0x1544A0
+#define FS_OFFSET_1310_EXFAT_SDMMC_WRAPPER_CONTROLLER_CLOSE 0x154530
+
+// Misc Data
+#define FS_OFFSET_1310_EXFAT_SD_MUTEX            0xE203E8
+#define FS_OFFSET_1310_EXFAT_NAND_MUTEX          0xE1B768
+#define FS_OFFSET_1310_EXFAT_ACTIVE_PARTITION    0xE1B7A8
+#define FS_OFFSET_1310_EXFAT_SDMMC_DAS_HANDLE    0xE03E18
+
+// NOPs
+#define FS_OFFSET_1310_EXFAT_SD_DAS_INIT         0x27744
+
+// Nintendo Paths
+#define FS_OFFSET_1310_EXFAT_NINTENDO_PATHS \
+{ \
+	{.opcode_reg = 3, .adrp_offset = 0x0006EBE0, .add_rel_offset = 0x00000004}, \
+	{.opcode_reg = 3, .adrp_offset = 0x0007BEEC, .add_rel_offset = 0x00000004}, \
+	{.opcode_reg = 4, .adrp_offset = 0x00082294, .add_rel_offset = 0x00000004}, \
+	{.opcode_reg = 4, .adrp_offset = 0x0009422C, .add_rel_offset = 0x00000004}, \
+	{.opcode_reg = 0, .adrp_offset = 0, .add_rel_offset = 0}, \
+}
+
+#endif // __FS_1310_EXFAT_H__

exosphere/program/source/secmon_exception_handler.cpp

@@ -22,7 +22,7 @@ namespace ams::secmon {
 
         constexpr inline uintptr_t PMC = MemoryRegionVirtualDevicePmc.GetAddress();
 
-        constinit std::atomic_bool g_is_locked = false;
+        constinit util::Atomic<bool> g_is_locked = false;
 
     }
 
@@ -72,7 +72,7 @@ namespace ams::secmon {
         }
 
         /* Acquire exclusive access to exception handling logic. */
-        if (!g_is_locked.exchange(true)) {
+        if (!g_is_locked.Exchange(true)) {
             /* Invoke the exception handler impl. */
             ExceptionHandlerImpl(lr, sp);
 

exosphere/program/source/smc/secmon_smc_se_lock.cpp

@@ -21,35 +21,47 @@ namespace ams::secmon::smc {
 
     namespace {
 
-        constinit std::atomic_bool g_is_locked = false;
+        constinit util::Atomic<bool> g_is_locked = false;
+
+        ALWAYS_INLINE bool TryLockSecurityEngineImpl() {
+            bool value = false;
+            return g_is_locked.CompareExchangeStrong(value, true);
+        }
+
+        ALWAYS_INLINE void UnlockSecurityEngineImpl() {
+            g_is_locked = false;
+        }
+
+        ALWAYS_INLINE bool IsSecurityEngineLockedImpl() {
+            return g_is_locked.Load();
+        }
 
     }
 
     bool TryLockSecurityEngine() {
-        bool value = false;
-        return g_is_locked.compare_exchange_strong(value, true);
+        return TryLockSecurityEngineImpl();
     }
 
     void UnlockSecurityEngine() {
-        g_is_locked = false;
+        return UnlockSecurityEngineImpl();
     }
 
     bool IsSecurityEngineLocked() {
-        return g_is_locked;
+        return IsSecurityEngineLockedImpl();
     }
 
     SmcResult LockSecurityEngineAndInvoke(SmcArguments &args, SmcHandler impl) {
         /* Try to lock the security engine. */
-        SMC_R_UNLESS(TryLockSecurityEngine(), Busy);
-        ON_SCOPE_EXIT { UnlockSecurityEngine(); };
+        SMC_R_UNLESS(TryLockSecurityEngineImpl(), Busy);
+        ON_SCOPE_EXIT { UnlockSecurityEngineImpl(); };
 
         return impl(args);
     }
 
     SmcResult LockSecurityEngineAndInvokeAsync(SmcArguments &args, SmcHandler impl, GetResultHandler result_handler) {
         /* Try to lock the security engine. */
-        SMC_R_UNLESS(TryLockSecurityEngine(), Busy);
-        auto se_guard = SCOPE_GUARD { UnlockSecurityEngine(); };
+        SMC_R_UNLESS(TryLockSecurityEngineImpl(), Busy);
+        auto se_guard = SCOPE_GUARD { UnlockSecurityEngineImpl(); };
 
         /* Try to start an async operation. */
         const u64 async_key = BeginAsyncOperation(result_handler);

fusee/program/source/fusee_overlay_manager.cpp

@@ -33,11 +33,11 @@ namespace ams::nxboot {
             u32 verif_hash;
             u32 store_hash;
             if (fuse::GetSocType() == fuse::SocType_Erista) {
-                result = fs::ReadFile(archive_file, __builtin_offsetof(ExternalPackage, ovl_mtc_erista), GetOverlayDestination(), sizeof(ExternalPackage{}.ovl_mtc_erista));
+                result = fs::ReadFile(archive_file, AMS_OFFSETOF(ExternalPackage, ovl_mtc_erista), GetOverlayDestination(), sizeof(ExternalPackage{}.ovl_mtc_erista));
                 verif_hash = reinterpret_cast<const u32 *>(GetOverlayDestination())[-2];
                 store_hash = reinterpret_cast<const u32 *>(GetOverlayDestination())[(sizeof(ExternalPackage{}.ovl_mtc_erista) / sizeof(u32)) - 1];
             } else /* if (fuse::GetSocType() == fuse::SocType_Mariko) */ {
-                result = fs::ReadFile(archive_file, __builtin_offsetof(ExternalPackage, ovl_mtc_mariko), GetOverlayDestination(), sizeof(ExternalPackage{}.ovl_mtc_mariko));
+                result = fs::ReadFile(archive_file, AMS_OFFSETOF(ExternalPackage, ovl_mtc_mariko), GetOverlayDestination(), sizeof(ExternalPackage{}.ovl_mtc_mariko));
                 verif_hash = reinterpret_cast<const u32 *>(GetOverlayDestination())[-1];
                 store_hash = reinterpret_cast<const u32 *>(GetOverlayDestination())[(sizeof(ExternalPackage{}.ovl_mtc_mariko) / sizeof(u32)) - 1];
             }

fusee/program/source/fusee_setup_horizon.cpp

@@ -146,7 +146,7 @@ namespace ams::nxboot {
                         /* Handle individual fields. */
                         for (const auto &entry : section.kv_list) {
                             if (std::strcmp(entry.key, "enabled") == 0) {
-                                enabled = entry.value[0] == '1';
+                                enabled = entry.value[0] != '0';
                             } else if (std::strcmp(entry.key, "id") == 0) {
                                 id = ParseHexInteger(entry.value);
                             } else if (std::strcmp(entry.key, "sector") == 0) {
@@ -270,7 +270,7 @@ namespace ams::nxboot {
             u8 *package2;
             size_t package2_size;
             {
-                constexpr s64 Package2Offset = __builtin_offsetof(pkg2::StorageLayout, package2_header);
+                constexpr s64 Package2Offset = AMS_OFFSETOF(pkg2::StorageLayout, package2_header);
 
                 pkg2::Package2Header header;
                 if (R_FAILED((result = ReadPackage2(Package2Offset, std::addressof(header), sizeof(header))))) {

fusee/program/source/fusee_stratosphere.cpp

@@ -150,6 +150,9 @@ namespace ams::nxboot {
             FsVersion_13_0_0,
             FsVersion_13_0_0_Exfat,
 
+            FsVersion_13_1_0,
+            FsVersion_13_1_0_Exfat,
+
             FsVersion_Count,
         };
 
@@ -215,6 +218,9 @@ namespace ams::nxboot {
 
             { 0x7D, 0x20, 0x05, 0x47, 0x17, 0x8A, 0x83, 0x6A }, /* FsVersion_13_0_0 */
             { 0x51, 0xEB, 0xFA, 0x9C, 0xCF, 0x66, 0xC0, 0x9E }, /* FsVersion_13_0_0_Exfat */
+
+            { 0x91, 0xBA, 0x65, 0xA2, 0x1C, 0x1D, 0x50, 0xAE }, /* FsVersion_13_1_0 */
+            { 0x76, 0x38, 0x27, 0xEE, 0x9C, 0x20, 0x7E, 0x5B }, /* FsVersion_13_1_0_Exfat */
         };
 
         const InitialProcessBinaryHeader *FindInitialProcessBinary(const pkg2::Package2Header *header, const u8 *data, ams::TargetFirmware target_firmware) {
@@ -603,6 +609,11 @@ namespace ams::nxboot {
                     AddPatch(fs_meta, 0x159119, NogcPatch0, sizeof(NogcPatch0));
                     AddPatch(fs_meta, 0x1426D0, NogcPatch1, sizeof(NogcPatch1));
                     break;
+                case FsVersion_13_1_0:
+                case FsVersion_13_1_0_Exfat:
+                    AddPatch(fs_meta, 0x1590B9, NogcPatch0, sizeof(NogcPatch0));
+                    AddPatch(fs_meta, 0x142670, NogcPatch1, sizeof(NogcPatch1));
+                    break;
                 default:
                     break;
             }

libraries/.gitrepo

@@ -6,7 +6,7 @@
 [subrepo]
 	remote = https://github.com/Atmosphere-NX/Atmosphere-libs
 	branch = master
-	commit = cf765c0946cc5c828364ae6bfccddc4041304f28
-	parent = 8634ea0f7c4f0e68adf2dfaaddc6ae1e225c4fc2
+	commit = ceff2f3712d66904747f77619cd6dc12a5dcea8b
+	parent = 6cf5205a28cfeb30930fd1cfd3c8369cd13edfa3
 	method = merge
 	cmdver = 0.4.1

libraries/config/arch/armv4t/arch.mk

@@ -0,0 +1,11 @@
+ifeq ($(strip $(DEVKITPRO)),)
+$(error "Please set DEVKITPRO in your environment. export DEVKITPRO=<path to>/devkitpro")
+endif
+
+include $(DEVKITPRO)/devkitARM/base_rules
+
+export ATMOSPHERE_DEFINES  += -DATMOSPHERE_ARCH_ARM_V4T
+export ATMOSPHERE_SETTINGS +=
+export ATMOSPHERE_CFLAGS   +=
+export ATMOSPHERE_CXXFLAGS +=
+export ATMOSPHERE_ASFLAGS  +=

libraries/config/arch/armv8a/arch.mk

@@ -0,0 +1,11 @@
+ifeq ($(strip $(DEVKITPRO)),)
+$(error "Please set DEVKITPRO in your environment. export DEVKITPRO=<path to>/devkitpro")
+endif
+
+include $(DEVKITPRO)/devkitA64/base_rules
+
+export ATMOSPHERE_DEFINES  += -DATMOSPHERE_ARCH_ARM_V8A
+export ATMOSPHERE_SETTINGS +=
+export ATMOSPHERE_CFLAGS   +=
+export ATMOSPHERE_CXXFLAGS +=
+export ATMOSPHERE_ASFLAGS  +=

libraries/config/board/qemu/virt/board.mk

@@ -0,0 +1,5 @@
+export ATMOSPHERE_DEFINES  += -DATMOSPHERE_BOARD_QEMU_VIRT -D__SWITCH__
+export ATMOSPHERE_SETTINGS +=
+export ATMOSPHERE_CFLAGS   +=
+export ATMOSPHERE_CXXFLAGS +=
+export ATMOSPHERE_ASFLAGS  +=

libraries/config/common.mk

@@ -23,7 +23,8 @@ export ATMOSPHERE_CFLAGS   := -Wall -ffunction-sections -fdata-sections -fno-str
                               -fno-asynchronous-unwind-tables -fno-unwind-tables -fno-stack-protector \
                               -Wno-format-truncation -Wno-format-zero-length -Wno-stringop-truncation
 
-export ATMOSPHERE_CXXFLAGS := -fno-rtti -fno-exceptions -std=gnu++20
+
+export ATMOSPHERE_CXXFLAGS := -fno-rtti -fno-exceptions -std=gnu++20 -Wno-invalid-offsetof
 export ATMOSPHERE_ASFLAGS  :=
 
 
@@ -38,6 +39,9 @@ export ATMOSPHERE_ARCH_NAME  := arm64
 export ATMOSPHERE_BOARD_NAME := nintendo_nx
 export ATMOSPHERE_OS_NAME    := horizon
 
+export ATMOSPHERE_SUB_ARCH_DIR  = armv8a
+export ATMOSPHERE_SUB_ARCH_NAME = armv8a
+
 export ATMOSPHERE_CPU_EXTENSIONS := arm_crypto_extension aarch64_crypto_extension
 else ifeq ($(ATMOSPHERE_CPU),arm7tdmi)
 export ATMOSPHERE_ARCH_DIR   := arm
@@ -48,9 +52,30 @@ export ATMOSPHERE_ARCH_NAME  := arm
 export ATMOSPHERE_BOARD_NAME := nintendo_nx
 export ATMOSPHERE_OS_NAME    := horizon
 
+export ATMOSPHERE_SUB_ARCH_DIR  = armv4t
+export ATMOSPHERE_SUB_ARCH_NAME = armv4t
+
 export ATMOSPHERE_CPU_EXTENSIONS :=
 endif
 
+else ifeq ($(ATMOSPHERE_BOARD),qemu-virt)
+
+
+ifeq ($(ATMOSPHERE_CPU),arm-cortex-a57)
+export ATMOSPHERE_ARCH_DIR   := arm64
+export ATMOSPHERE_BOARD_DIR  := qemu/virt
+export ATMOSPHERE_OS_DIR     := horizon
+
+export ATMOSPHERE_ARCH_NAME  := arm64
+export ATMOSPHERE_BOARD_NAME := qemu_virt
+export ATMOSPHERE_OS_NAME    := horizon
+
+export ATMOSPHERE_SUB_ARCH_DIR  = armv8a
+export ATMOSPHERE_SUB_ARCH_NAME = armv8a
+
+export ATMOSPHERE_CPU_EXTENSIONS := arm_crypto_extension aarch64_crypto_extension
+endif
+
 endif
 
 ifeq ($(ATMOSPHERE_CPU),arm-cortex-a57)
@@ -69,14 +94,27 @@ export ATMOSPHERE_BOARD_MAKE_DIR := $(ATMOSPHERE_CONFIG_MAKE_DIR)/board/$(ATMOSP
 export ATMOSPHERE_OS_MAKE_DIR    := $(ATMOSPHERE_CONFIG_MAKE_DIR)/os/$(ATMOSPHERE_OS_DIR)
 export ATMOSPHERE_CPU_MAKE_DIR   := $(ATMOSPHERE_ARCH_MAKE_DIR)/cpu/$(ATMOSPHERE_CPU_DIR)
 
+ifneq ($(strip $(ATMOSPHERE_SUB_ARCH_NAME)),)
+export ATMOSPHERE_LIBRARY_DIR := lib_$(ATMOSPHERE_BOARD_NAME)_$(ATMOSPHERE_ARCH_NAME)_$(ATMOSPHERE_SUB_ARCH_NAME)
+export ATMOSPHERE_BUILD_DIR := build_$(ATMOSPHERE_BOARD_NAME)_$(ATMOSPHERE_ARCH_NAME)_$(ATMOSPHERE_SUB_ARCH_NAME)
+else
 export ATMOSPHERE_LIBRARY_DIR := lib_$(ATMOSPHERE_BOARD_NAME)_$(ATMOSPHERE_ARCH_NAME)
 export ATMOSPHERE_BUILD_DIR := build_$(ATMOSPHERE_BOARD_NAME)_$(ATMOSPHERE_ARCH_NAME)
+endif
 
 include $(ATMOSPHERE_ARCH_MAKE_DIR)/arch.mk
 include $(ATMOSPHERE_BOARD_MAKE_DIR)/board.mk
 include $(ATMOSPHERE_OS_MAKE_DIR)/os.mk
 include $(ATMOSPHERE_CPU_MAKE_DIR)/cpu.mk
 
+
+ifneq ($(strip $(ATMOSPHERE_SUB_ARCH_NAME)),)
+export ATMOSPHERE_SUB_ARCH_MAKE_DIR  := $(ATMOSPHERE_CONFIG_MAKE_DIR)/arch/$(ATMOSPHERE_SUB_ARCH_DIR)
+
+include $(ATMOSPHERE_SUB_ARCH_MAKE_DIR)/arch.mk
+endif
+
+
 #---------------------------------------------------------------------------------
 # get atmosphere git revision information
 #---------------------------------------------------------------------------------
@@ -109,12 +147,20 @@ DATA         := data
 INCLUDES     := include
 
 GENERAL_SOURCE_DIRS=$1 $(foreach d,$(filter-out $1/arch $1/board $1/os $1/cpu $1,$(wildcard $1/*)),$(if $(wildcard $d/.),$(filter-out $d,$(call GENERAL_SOURCE_DIRS,$d)) $d,))
+
 SPECIFIC_SOURCE_DIRS=$(if $(wildcard $1/$2/$3/.*),$1/$2/$3 $(call DIR_WILDCARD,$1/$2/$3),$(if $(wildcard $1/$2/generic/.*), $1/$2/generic $(call DIR_WILDCARD,$1/$2/generic),))
+
+ifneq ($(strip $(ATMOSPHERE_SUB_ARCH_NAME)),)
+SPECIFIC_SOURCE_DIRS_ARCH=$(if $(wildcard $1/$2/$3/.*),$1/$2/$3 $(call DIR_WILDCARD,$1/$2/$3),$(if $(wildcard $1/$2/$4/.*),$1/$2/$4 $(call DIR_WILDCARD,$1/$2/$4),$(if $(wildcard $1/$2/generic/.*), $1/$2/generic $(call DIR_WILDCARD,$1/$2/generic),)))
+else
+SPECIFIC_SOURCE_DIRS_ARCH=$(if $(wildcard $1/$2/$3/.*),$1/$2/$3 $(call DIR_WILDCARD,$1/$2/$3),$(if $(wildcard $1/$2/generic/.*), $1/$2/generic $(call DIR_WILDCARD,$1/$2/generic),))
+endif
+
 UNFILTERED_SOURCE_DIRS=$1 $(foreach d,$(wildcard $1/*),$(if $(wildcard $d/.),$(call DIR_WILDCARD,$d) $d,))
 
 ALL_SOURCE_DIRS=$(foreach d,$(call GENERAL_SOURCE_DIRS,$1), \
                     $d \
-                    $(call SPECIFIC_SOURCE_DIRS,$d,arch,$(ATMOSPHERE_ARCH_DIR)) \
+                    $(call SPECIFIC_SOURCE_DIRS_ARCH,$d,arch,$(ATMOSPHERE_ARCH_DIR),$(ATMOSPHERE_SUB_ARCH_DIR)) \
                     $(call SPECIFIC_SOURCE_DIRS,$d,board,$(ATMOSPHERE_BOARD_DIR)) \
                     $(call SPECIFIC_SOURCE_DIRS,$d,os,$(ATMOSPHERE_OS_DIR)) \
                     $(call SPECIFIC_SOURCE_DIRS,$d,cpu,$(ATMOSPHERE_ARCH_DIR)/$(ATMOSPHERE_CPU_DIR)) \
@@ -131,7 +177,7 @@ FIND_SOURCE_FILES=$(foreach dir,$1,$(filter-out $(notdir $(wildcard $(dir)/*.arc
                                                     $(notdir $(wildcard $(dir)/*.os.*.$2)) \
                                                     $(notdir $(wildcard $(dir)/.cpu.*.$2)), \
                                                 $(notdir $(wildcard $(dir)/*.$2)))) \
-                  $(foreach dir,$1,$(call FIND_SPECIFIC_SOURCE_FILES,$(dir),arch,$(ATMOSPHERE_ARCH_NAME),$2)) \
+                  $(foreach dir,$1,$(call FIND_SPECIFIC_SOURCE_FILES_EX,$(dir),arch,$(ATMOSPHERE_ARCH_NAME) $(ATMOSPHERE_SUB_ARCH_NAME),$2)) \
                   $(foreach dir,$1,$(call FIND_SPECIFIC_SOURCE_FILES,$(dir),board,$(ATMOSPHERE_BOARD_NAME),$2)) \
                   $(foreach dir,$1,$(call FIND_SPECIFIC_SOURCE_FILES,$(dir),os,$(ATMOSPHERE_OS_NAME),$2)) \
                   $(foreach dir,$1,$(call FIND_SPECIFIC_SOURCE_FILES_EX,$(dir),cpu,$(ATMOSPHERE_CPU_NAME) $(ATMOSPHERE_CPU_EXTENSIONS),$2))

libraries/libexosphere/include/exosphere/pkg1/pkg1_bootloader_parameters.hpp

@@ -55,9 +55,9 @@ namespace ams::pkg1 {
     static_assert(util::is_pod<SecureMonitorParameters>::value);
     static_assert(sizeof(SecureMonitorParameters) == 0x200);
 
-    static_assert(offsetof(SecureMonitorParameters, bct_params)       == 0x10);
-    static_assert(offsetof(SecureMonitorParameters, bootloader_state) == 0xF8);
-    static_assert(offsetof(SecureMonitorParameters, secmon_state) == 0xFC);
+    static_assert(AMS_OFFSETOF(SecureMonitorParameters, bct_params)       == 0x10);
+    static_assert(AMS_OFFSETOF(SecureMonitorParameters, bootloader_state) == 0xF8);
+    static_assert(AMS_OFFSETOF(SecureMonitorParameters, secmon_state) == 0xFC);
 
     enum BootloaderAttribute {
         BootloaderAttribute_None         = (0u << 0),

libraries/libexosphere/include/exosphere/secmon/secmon_volatile_context.hpp

@@ -128,10 +128,10 @@ namespace ams::secmon {
 
     }
 
-    constexpr inline const Address WarmbootStackAddress   = MemoryRegionVirtualTzramVolatileData.GetAddress() + offsetof(VolatileData, warmboot_stack)    + sizeof(VolatileData::warmboot_stack);
-    constexpr inline const Address Core012SmcStackAddress = MemoryRegionVirtualTzramVolatileData.GetAddress() + offsetof(VolatileData, core012_smc_stack) + sizeof(VolatileData::core012_smc_stack);
+    constexpr inline const Address WarmbootStackAddress   = MemoryRegionVirtualTzramVolatileData.GetAddress() + AMS_OFFSETOF(VolatileData, warmboot_stack)    + sizeof(VolatileData::warmboot_stack);
+    constexpr inline const Address Core012SmcStackAddress = MemoryRegionVirtualTzramVolatileData.GetAddress() + AMS_OFFSETOF(VolatileData, core012_smc_stack) + sizeof(VolatileData::core012_smc_stack);
 
-    constexpr inline const Address Core0ExceptionStackAddress = MemoryRegionVirtualTzramVolatileData.GetAddress() + offsetof(VolatileData, core_exception_stacks) + CoreExceptionStackSize;
+    constexpr inline const Address Core0ExceptionStackAddress = MemoryRegionVirtualTzramVolatileData.GetAddress() + AMS_OFFSETOF(VolatileData, core_exception_stacks) + CoreExceptionStackSize;
     constexpr inline const Address Core1ExceptionStackAddress = Core0ExceptionStackAddress + CoreExceptionStackSize;
     constexpr inline const Address Core2ExceptionStackAddress = Core1ExceptionStackAddress + CoreExceptionStackSize;
 

libraries/libexosphere/source/gic/gic_api.cpp

@@ -191,27 +191,27 @@ namespace ams::gic {
     }
 
     void SetPriority(int interrupt_id, int priority) {
-        ReadWrite(g_distributor_address + offsetof(GicDistributor, ipriorityr), BITSIZEOF(u8), interrupt_id, priority);
+        ReadWrite(g_distributor_address + AMS_OFFSETOF(GicDistributor, ipriorityr), BITSIZEOF(u8), interrupt_id, priority);
     }
 
     void SetInterruptGroup(int interrupt_id, int group) {
-        ReadWrite(g_distributor_address + offsetof(GicDistributor, igroupr), 1, interrupt_id, group);
+        ReadWrite(g_distributor_address + AMS_OFFSETOF(GicDistributor, igroupr), 1, interrupt_id, group);
     }
 
     void SetEnable(int interrupt_id, bool enable) {
-        Write(g_distributor_address + offsetof(GicDistributor, isenabler), 1, interrupt_id, enable);
+        Write(g_distributor_address + AMS_OFFSETOF(GicDistributor, isenabler), 1, interrupt_id, enable);
     }
 
     void SetSpiTargetCpu(int interrupt_id, u32 cpu_mask) {
-        ReadWrite(g_distributor_address + offsetof(GicDistributor, itargetsr), BITSIZEOF(u8), interrupt_id, cpu_mask);
+        ReadWrite(g_distributor_address + AMS_OFFSETOF(GicDistributor, itargetsr), BITSIZEOF(u8), interrupt_id, cpu_mask);
     }
 
     void SetSpiMode(int interrupt_id, InterruptMode mode) {
-        ReadWrite(g_distributor_address + offsetof(GicDistributor, icfgr), 2, interrupt_id, static_cast<u32>(mode) << 1);
+        ReadWrite(g_distributor_address + AMS_OFFSETOF(GicDistributor, icfgr), 2, interrupt_id, static_cast<u32>(mode) << 1);
     }
 
     void SetPending(int interrupt_id) {
-        Write(g_distributor_address + offsetof(GicDistributor, ispendr), 1, interrupt_id, 1);
+        Write(g_distributor_address + AMS_OFFSETOF(GicDistributor, ispendr), 1, interrupt_id, 1);
     }
 
     int GetInterruptRequestId() {

libraries/libmesosphere/Makefile

@@ -104,6 +104,18 @@ $(eval $(call ATMOSPHERE_ADD_TARGET, audit, $(TARGET)_audit.a, \
 	ATMOSPHERE_BUILD_SETTINGS="-DMESOSPHERE_BUILD_FOR_AUDITING" \
 ))
 
+$(eval $(call ATMOSPHERE_ADD_TARGET, qemu_virt_release, $(TARGET)_qemu_virt.a, \
+	ATMOSPHERE_BUILD_SETTINGS="" \
+))
+
+$(eval $(call ATMOSPHERE_ADD_TARGET, qemu_virt_debug, $(TARGET)_qemu_virt_debug.a, \
+	ATMOSPHERE_BUILD_SETTINGS="-DMESOSPHERE_BUILD_FOR_DEBUGGING" \
+))
+
+$(eval $(call ATMOSPHERE_ADD_TARGET, qemu_virt_audit, $(TARGET)_qemu_virt_audit.a, \
+	ATMOSPHERE_BUILD_SETTINGS="-DMESOSPHERE_BUILD_FOR_AUDITING" \
+))
+
 #---------------------------------------------------------------------------------
 
 -include $(ATMOSPHERE_BOARD_NAME)_$(ATMOSPHERE_ARCH_NAME).mk
@@ -142,8 +154,6 @@ $(OFILES_SRC)	: $(HFILES_BIN)
 
 kern_libc_generic.o: CFLAGS += -fno-builtin
 
-kern_k_auto_object.o kern_k_debug_base_process_holder.o: CXXFLAGS += -fno-lto
-
 #---------------------------------------------------------------------------------
 %_bin.h %.bin.o	:	%.bin
 #---------------------------------------------------------------------------------

libraries/libmesosphere/include/mesosphere.hpp

@@ -90,3 +90,7 @@
 
 /* Main functionality. */
 #include <mesosphere/kern_main.hpp>
+
+/* Deferred includes. */
+#include <mesosphere/kern_k_auto_object_impls.hpp>
+#include <mesosphere/kern_k_scheduler_impls.hpp>

libraries/libmesosphere/include/mesosphere/arch/arm/kern_generic_interrupt_controller.hpp

@@ -248,8 +248,9 @@ namespace ams::kern::arch::arm {
                 return id;
             }
         private:
-            static constexpr size_t PriorityShift = BITSIZEOF(u8) - __builtin_ctz(NumPriorityLevels);
+            static constexpr size_t PriorityShift = BITSIZEOF(u8) - util::CountTrailingZeros(NumPriorityLevels);
             static_assert(PriorityShift < BITSIZEOF(u8));
+            static_assert(util::IsPowerOfTwo(NumPriorityLevels));
 
             static constexpr ALWAYS_INLINE u8 ToGicPriorityValue(s32 level) {
                 return (level << PriorityShift) | ((1 << PriorityShift) - 1);

libraries/libmesosphere/include/mesosphere/arch/arm64/init/kern_k_init_arguments.hpp

@@ -35,17 +35,17 @@ namespace ams::kern::init {
     static_assert(alignof(KInitArguments) == util::CeilingPowerOfTwo(INIT_ARGUMENTS_SIZE));
     static_assert(sizeof(KInitArguments) == std::max(INIT_ARGUMENTS_SIZE, util::CeilingPowerOfTwo(INIT_ARGUMENTS_SIZE)));
 
-    static_assert(__builtin_offsetof(KInitArguments, ttbr0)           == INIT_ARGUMENTS_TTBR0);
-    static_assert(__builtin_offsetof(KInitArguments, ttbr1)           == INIT_ARGUMENTS_TTBR1);
-    static_assert(__builtin_offsetof(KInitArguments, tcr)             == INIT_ARGUMENTS_TCR);
-    static_assert(__builtin_offsetof(KInitArguments, mair)            == INIT_ARGUMENTS_MAIR);
-    static_assert(__builtin_offsetof(KInitArguments, cpuactlr)        == INIT_ARGUMENTS_CPUACTLR);
-    static_assert(__builtin_offsetof(KInitArguments, cpuectlr)        == INIT_ARGUMENTS_CPUECTLR);
-    static_assert(__builtin_offsetof(KInitArguments, sctlr)           == INIT_ARGUMENTS_SCTLR);
-    static_assert(__builtin_offsetof(KInitArguments, sp)              == INIT_ARGUMENTS_SP);
-    static_assert(__builtin_offsetof(KInitArguments, entrypoint)      == INIT_ARGUMENTS_ENTRYPOINT);
-    static_assert(__builtin_offsetof(KInitArguments, argument)        == INIT_ARGUMENTS_ARGUMENT);
-    static_assert(__builtin_offsetof(KInitArguments, setup_function)  == INIT_ARGUMENTS_SETUP_FUNCTION);
-    static_assert(__builtin_offsetof(KInitArguments, exception_stack) == INIT_ARGUMENTS_EXCEPTION_STACK);
+    static_assert(AMS_OFFSETOF(KInitArguments, ttbr0)           == INIT_ARGUMENTS_TTBR0);
+    static_assert(AMS_OFFSETOF(KInitArguments, ttbr1)           == INIT_ARGUMENTS_TTBR1);
+    static_assert(AMS_OFFSETOF(KInitArguments, tcr)             == INIT_ARGUMENTS_TCR);
+    static_assert(AMS_OFFSETOF(KInitArguments, mair)            == INIT_ARGUMENTS_MAIR);
+    static_assert(AMS_OFFSETOF(KInitArguments, cpuactlr)        == INIT_ARGUMENTS_CPUACTLR);
+    static_assert(AMS_OFFSETOF(KInitArguments, cpuectlr)        == INIT_ARGUMENTS_CPUECTLR);
+    static_assert(AMS_OFFSETOF(KInitArguments, sctlr)           == INIT_ARGUMENTS_SCTLR);
+    static_assert(AMS_OFFSETOF(KInitArguments, sp)              == INIT_ARGUMENTS_SP);
+    static_assert(AMS_OFFSETOF(KInitArguments, entrypoint)      == INIT_ARGUMENTS_ENTRYPOINT);
+    static_assert(AMS_OFFSETOF(KInitArguments, argument)        == INIT_ARGUMENTS_ARGUMENT);
+    static_assert(AMS_OFFSETOF(KInitArguments, setup_function)  == INIT_ARGUMENTS_SETUP_FUNCTION);
+    static_assert(AMS_OFFSETOF(KInitArguments, exception_stack) == INIT_ARGUMENTS_EXCEPTION_STACK);
 
 }

libraries/libmesosphere/include/mesosphere/arch/arm64/init/kern_k_init_page_table.hpp

@@ -34,19 +34,22 @@ namespace ams::kern::arch::arm64::init {
         }
     }
 
-    class KInitialPageTable {
+    /* NOTE: Nintendo uses virtual functions, rather than a concept + template. */
+    template<typename T>
+    concept IsInitialPageAllocator = requires (T &t, KPhysicalAddress phys_addr, size_t size) {
+        { t.Allocate(size) }        -> std::same_as<KPhysicalAddress>;
+        { t.Free(phys_addr, size) } -> std::same_as<void>;
+    };
+
+    template<IsInitialPageAllocator _PageAllocator>
+    class KInitialPageTableTemplate {
         public:
-            class IPageAllocator {
-                public:
-                    virtual KPhysicalAddress Allocate(size_t size) = 0;
-                    virtual void Free(KPhysicalAddress phys_addr, size_t size) = 0;
-            };
+            using PageAllocator = _PageAllocator;
         private:
             KPhysicalAddress m_l1_tables[2];
             u32 m_num_entries[2];
-
         public:
-            KInitialPageTable(KVirtualAddress start_address, KVirtualAddress end_address, IPageAllocator &allocator) {
+            KInitialPageTableTemplate(KVirtualAddress start_address, KVirtualAddress end_address, PageAllocator &allocator) {
                 /* Set tables. */
                 m_l1_tables[0] = AllocateNewPageTable(allocator);
                 m_l1_tables[1] = AllocateNewPageTable(allocator);
@@ -56,7 +59,7 @@ namespace ams::kern::arch::arm64::init {
                 m_num_entries[1] = ((end_address / L1BlockSize) & (MaxPageTableEntries - 1)) - ((start_address / L1BlockSize) & (MaxPageTableEntries - 1)) + 1;
             }
 
-            KInitialPageTable() {
+            KInitialPageTableTemplate() {
                 /* Set tables. */
                 m_l1_tables[0] = util::AlignDown(cpu::GetTtbr0El1(), PageSize);
                 m_l1_tables[1] = util::AlignDown(cpu::GetTtbr1El1(), PageSize);
@@ -95,7 +98,7 @@ namespace ams::kern::arch::arm64::init {
                 return l3_table + ((GetInteger(address) / L3BlockSize) & (MaxPageTableEntries - 1));
             }
 
-            static ALWAYS_INLINE KPhysicalAddress AllocateNewPageTable(IPageAllocator &allocator) {
+            static ALWAYS_INLINE KPhysicalAddress AllocateNewPageTable(PageAllocator &allocator) {
                 auto address = allocator.Allocate(PageSize);
                 ClearNewPageTable(address);
                 return address;
@@ -276,7 +279,7 @@ namespace ams::kern::arch::arm64::init {
 
                 /* Invalidate the entire tlb. */
                 cpu::DataSynchronizationBarrierInnerShareable();
-                cpu::InvalidateEntireTlb();
+                cpu::InvalidateEntireTlbInnerShareable();
 
                 /* Copy data, if we should. */
                 const u64 negative_block_size_for_mask = static_cast<u64>(-static_cast<s64>(block_size));
@@ -323,7 +326,7 @@ namespace ams::kern::arch::arm64::init {
                 }
             }
         public:
-            void NOINLINE Map(KVirtualAddress virt_addr, size_t size, KPhysicalAddress phys_addr, const PageTableEntry &attr, IPageAllocator &allocator) {
+            void NOINLINE Map(KVirtualAddress virt_addr, size_t size, KPhysicalAddress phys_addr, const PageTableEntry &attr, PageAllocator &allocator) {
                 /* Ensure that addresses and sizes are page aligned. */
                 MESOSPHERE_INIT_ABORT_UNLESS(util::IsAligned(GetInteger(virt_addr),    PageSize));
                 MESOSPHERE_INIT_ABORT_UNLESS(util::IsAligned(GetInteger(phys_addr),    PageSize));
@@ -347,7 +350,6 @@ namespace ams::kern::arch::arm64::init {
                     /* If we don't already have an L2 table, we need to make a new one. */
                     if (!l1_entry->IsTable()) {
                         KPhysicalAddress new_table = AllocateNewPageTable(allocator);
-                        ClearNewPageTable(new_table);
                         *l1_entry = L1PageTableEntry(PageTableEntry::TableTag{}, new_table, attr.IsPrivilegedExecuteNever());
                         cpu::DataSynchronizationBarrierInnerShareable();
                     }
@@ -358,12 +360,12 @@ namespace ams::kern::arch::arm64::init {
                     if (util::IsAligned(GetInteger(virt_addr), L2ContiguousBlockSize) && util::IsAligned(GetInteger(phys_addr), L2ContiguousBlockSize) && size >= L2ContiguousBlockSize) {
                         for (size_t i = 0; i < L2ContiguousBlockSize / L2BlockSize; i++) {
                             l2_entry[i] = L2PageTableEntry(PageTableEntry::BlockTag{}, phys_addr, attr, PageTableEntry::SoftwareReservedBit_None, true);
-                            cpu::DataSynchronizationBarrierInnerShareable();
 
                             virt_addr += L2BlockSize;
                             phys_addr += L2BlockSize;
                             size      -= L2BlockSize;
                         }
+                        cpu::DataSynchronizationBarrierInnerShareable();
                         continue;
                     }
 
@@ -381,7 +383,6 @@ namespace ams::kern::arch::arm64::init {
                     /* If we don't already have an L3 table, we need to make a new one. */
                     if (!l2_entry->IsTable()) {
                         KPhysicalAddress new_table = AllocateNewPageTable(allocator);
-                        ClearNewPageTable(new_table);
                         *l2_entry = L2PageTableEntry(PageTableEntry::TableTag{}, new_table, attr.IsPrivilegedExecuteNever());
                         cpu::DataSynchronizationBarrierInnerShareable();
                     }
@@ -392,12 +393,12 @@ namespace ams::kern::arch::arm64::init {
                     if (util::IsAligned(GetInteger(virt_addr), L3ContiguousBlockSize) && util::IsAligned(GetInteger(phys_addr), L3ContiguousBlockSize) && size >= L3ContiguousBlockSize) {
                         for (size_t i = 0; i < L3ContiguousBlockSize / L3BlockSize; i++) {
                             l3_entry[i] = L3PageTableEntry(PageTableEntry::BlockTag{}, phys_addr, attr, PageTableEntry::SoftwareReservedBit_None, true);
-                            cpu::DataSynchronizationBarrierInnerShareable();
 
                             virt_addr += L3BlockSize;
                             phys_addr += L3BlockSize;
                             size      -= L3BlockSize;
                         }
+                        cpu::DataSynchronizationBarrierInnerShareable();
                         continue;
                     }
 
@@ -700,10 +701,9 @@ namespace ams::kern::arch::arm64::init {
                 this->PhysicallyRandomize(virt_addr, size, L3BlockSize, do_copy);
                 cpu::StoreEntireCacheForInit();
             }
-
     };
 
-    class KInitialPageAllocator final : public KInitialPageTable::IPageAllocator {
+    class KInitialPageAllocator final {
         private:
             static constexpr inline size_t FreeUnitSize = BITSIZEOF(u64) * PageSize;
             struct FreeListEntry {
@@ -807,11 +807,11 @@ namespace ams::kern::arch::arm64::init {
                 }
             }
 
-            virtual KPhysicalAddress Allocate(size_t size) override {
+            KPhysicalAddress Allocate(size_t size) {
                 return this->Allocate(size, size);
             }
 
-            virtual void Free(KPhysicalAddress phys_addr, size_t size) override {
+            void Free(KPhysicalAddress phys_addr, size_t size) {
                 auto **prev_next = std::addressof(m_state.free_head);
                 auto *new_chunk  = reinterpret_cast<FreeListEntry *>(GetInteger(phys_addr));
                 if (auto *cur = m_state.free_head; cur != nullptr) {
@@ -863,5 +863,8 @@ namespace ams::kern::arch::arm64::init {
                 *prev_next = new_chunk;
             }
     };
+    static_assert(IsInitialPageAllocator<KInitialPageAllocator>);
+
+    using KInitialPageTable = KInitialPageTableTemplate<KInitialPageAllocator>;
 
 }

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_assembly_offsets.h

@@ -19,6 +19,9 @@
 /* TODO: Different header for this? */
 #define AMS_KERN_NUM_SUPERVISOR_CALLS 0xC0
 
+/* ams::kern::KThread, https://github.com/Atmosphere-NX/Atmosphere/blob/master/libraries/libmesosphere/include/mesosphere/kern_k_thread.hpp */
+#define THREAD_THREAD_CONTEXT 0xD0
+
 /* ams::kern::KThread::StackParameters, https://github.com/Atmosphere-NX/Atmosphere/blob/master/libraries/libmesosphere/include/mesosphere/kern_k_thread.hpp */
 #define THREAD_STACK_PARAMETERS_SIZE                    0x30
 #define THREAD_STACK_PARAMETERS_SVC_PERMISSION          0x00

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_cpu.hpp

@@ -30,6 +30,8 @@ namespace ams::kern::arch::arm64::cpu {
 
 #if defined(ATMOSPHERE_BOARD_NINTENDO_NX)
     constexpr inline size_t NumCores = 4;
+#elif defined(ATMOSPHERE_BOARD_QEMU_VIRT)
+    constexpr inline size_t NumCores = 4;
 #else
     #error "Unknown Board for cpu::NumCores"
 #endif
@@ -50,10 +52,19 @@ namespace ams::kern::arch::arm64::cpu {
         __asm__ __volatile__("dmb sy" ::: "memory");
     }
 
+    ALWAYS_INLINE void DataMemoryBarrierInnerShareable() {
+        __asm__ __volatile__("dmb ish" ::: "memory");
+    }
+
     ALWAYS_INLINE void InstructionMemoryBarrier() {
         __asm__ __volatile__("isb" ::: "memory");
     }
 
+    ALWAYS_INLINE void EnsureInstructionConsistencyInnerShareable() {
+        DataSynchronizationBarrierInnerShareable();
+        InstructionMemoryBarrier();
+    }
+
     ALWAYS_INLINE void EnsureInstructionConsistency() {
         DataSynchronizationBarrier();
         InstructionMemoryBarrier();
@@ -171,7 +182,6 @@ namespace ams::kern::arch::arm64::cpu {
     NOINLINE void SynchronizeAllCores();
 
     /* Cache management helpers. */
-    void ClearPageToZeroImpl(void *);
     void StoreEntireCacheForInit();
     void FlushEntireCacheForInit();
 
@@ -184,10 +194,16 @@ namespace ams::kern::arch::arm64::cpu {
 
     void InvalidateEntireInstructionCache();
 
-    ALWAYS_INLINE void ClearPageToZero(void *page) {
+    ALWAYS_INLINE void ClearPageToZero(void * const page) {
         MESOSPHERE_ASSERT(util::IsAligned(reinterpret_cast<uintptr_t>(page), PageSize));
         MESOSPHERE_ASSERT(page != nullptr);
-        ClearPageToZeroImpl(page);
+
+        uintptr_t cur = reinterpret_cast<uintptr_t>(__builtin_assume_aligned(page, PageSize));
+        const uintptr_t last = cur + PageSize - DataCacheLineSize;
+
+        for (/* ... */; cur <= last; cur += DataCacheLineSize) {
+            __asm__ __volatile__("dc zva, %[cur]" :: [cur]"r"(cur) : "memory");
+        }
     }
 
     ALWAYS_INLINE void InvalidateTlbByAsid(u32 asid) {
@@ -207,6 +223,11 @@ namespace ams::kern::arch::arm64::cpu {
         EnsureInstructionConsistency();
     }
 
+    ALWAYS_INLINE void InvalidateEntireTlbInnerShareable() {
+        __asm__ __volatile__("tlbi vmalle1is" ::: "memory");
+        EnsureInstructionConsistencyInnerShareable();
+    }
+
     ALWAYS_INLINE void InvalidateEntireTlbDataOnly() {
         __asm__ __volatile__("tlbi vmalle1is" ::: "memory");
         DataSynchronizationBarrier();

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_k_debug.hpp

@@ -34,8 +34,9 @@ namespace ams::kern::arch::arm64 {
 
             static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
         public:
-            virtual Result GetThreadContextImpl(ams::svc::ThreadContext *out, KThread *thread, u32 context_flags) override;
-            virtual Result SetThreadContextImpl(const ams::svc::ThreadContext &ctx, KThread *thread, u32 context_flags) override;
+            /* NOTE: These are virtual in Nintendo's kernel. */
+            Result GetThreadContextImpl(ams::svc::ThreadContext *out, KThread *thread, u32 context_flags);
+            Result SetThreadContextImpl(const ams::svc::ThreadContext &ctx, KThread *thread, u32 context_flags);
         private:
             Result GetFpuContext(ams::svc::ThreadContext *out, KThread *thread, u32 context_flags);
             Result SetFpuContext(const ams::svc::ThreadContext &ctx, KThread *thread, u32 context_flags);

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_k_exception_context.hpp

@@ -45,40 +45,40 @@ namespace ams::kern::arch::arm64 {
     };
     static_assert(sizeof(KExceptionContext) == EXCEPTION_CONTEXT_SIZE);
 
-    static_assert(__builtin_offsetof(KExceptionContext, x[ 0]) == EXCEPTION_CONTEXT_X0);
-    static_assert(__builtin_offsetof(KExceptionContext, x[ 1]) == EXCEPTION_CONTEXT_X1);
-    static_assert(__builtin_offsetof(KExceptionContext, x[ 2]) == EXCEPTION_CONTEXT_X2);
-    static_assert(__builtin_offsetof(KExceptionContext, x[ 3]) == EXCEPTION_CONTEXT_X3);
-    static_assert(__builtin_offsetof(KExceptionContext, x[ 4]) == EXCEPTION_CONTEXT_X4);
-    static_assert(__builtin_offsetof(KExceptionContext, x[ 5]) == EXCEPTION_CONTEXT_X5);
-    static_assert(__builtin_offsetof(KExceptionContext, x[ 6]) == EXCEPTION_CONTEXT_X6);
-    static_assert(__builtin_offsetof(KExceptionContext, x[ 7]) == EXCEPTION_CONTEXT_X7);
-    static_assert(__builtin_offsetof(KExceptionContext, x[ 8]) == EXCEPTION_CONTEXT_X8);
-    static_assert(__builtin_offsetof(KExceptionContext, x[ 9]) == EXCEPTION_CONTEXT_X9);
-    static_assert(__builtin_offsetof(KExceptionContext, x[10]) == EXCEPTION_CONTEXT_X10);
-    static_assert(__builtin_offsetof(KExceptionContext, x[11]) == EXCEPTION_CONTEXT_X11);
-    static_assert(__builtin_offsetof(KExceptionContext, x[12]) == EXCEPTION_CONTEXT_X12);
-    static_assert(__builtin_offsetof(KExceptionContext, x[13]) == EXCEPTION_CONTEXT_X13);
-    static_assert(__builtin_offsetof(KExceptionContext, x[14]) == EXCEPTION_CONTEXT_X14);
-    static_assert(__builtin_offsetof(KExceptionContext, x[15]) == EXCEPTION_CONTEXT_X15);
-    static_assert(__builtin_offsetof(KExceptionContext, x[16]) == EXCEPTION_CONTEXT_X16);
-    static_assert(__builtin_offsetof(KExceptionContext, x[17]) == EXCEPTION_CONTEXT_X17);
-    static_assert(__builtin_offsetof(KExceptionContext, x[18]) == EXCEPTION_CONTEXT_X18);
-    static_assert(__builtin_offsetof(KExceptionContext, x[19]) == EXCEPTION_CONTEXT_X19);
-    static_assert(__builtin_offsetof(KExceptionContext, x[20]) == EXCEPTION_CONTEXT_X20);
-    static_assert(__builtin_offsetof(KExceptionContext, x[21]) == EXCEPTION_CONTEXT_X21);
-    static_assert(__builtin_offsetof(KExceptionContext, x[22]) == EXCEPTION_CONTEXT_X22);
-    static_assert(__builtin_offsetof(KExceptionContext, x[23]) == EXCEPTION_CONTEXT_X23);
-    static_assert(__builtin_offsetof(KExceptionContext, x[24]) == EXCEPTION_CONTEXT_X24);
-    static_assert(__builtin_offsetof(KExceptionContext, x[25]) == EXCEPTION_CONTEXT_X25);
-    static_assert(__builtin_offsetof(KExceptionContext, x[26]) == EXCEPTION_CONTEXT_X26);
-    static_assert(__builtin_offsetof(KExceptionContext, x[27]) == EXCEPTION_CONTEXT_X27);
-    static_assert(__builtin_offsetof(KExceptionContext, x[28]) == EXCEPTION_CONTEXT_X28);
-    static_assert(__builtin_offsetof(KExceptionContext, x[29]) == EXCEPTION_CONTEXT_X29);
-    static_assert(__builtin_offsetof(KExceptionContext, x[30]) == EXCEPTION_CONTEXT_X30);
-    static_assert(__builtin_offsetof(KExceptionContext, sp)    == EXCEPTION_CONTEXT_SP);
-    static_assert(__builtin_offsetof(KExceptionContext, pc)    == EXCEPTION_CONTEXT_PC);
-    static_assert(__builtin_offsetof(KExceptionContext, psr)   == EXCEPTION_CONTEXT_PSR);
-    static_assert(__builtin_offsetof(KExceptionContext, tpidr) == EXCEPTION_CONTEXT_TPIDR);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[ 0]) == EXCEPTION_CONTEXT_X0);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[ 1]) == EXCEPTION_CONTEXT_X1);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[ 2]) == EXCEPTION_CONTEXT_X2);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[ 3]) == EXCEPTION_CONTEXT_X3);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[ 4]) == EXCEPTION_CONTEXT_X4);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[ 5]) == EXCEPTION_CONTEXT_X5);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[ 6]) == EXCEPTION_CONTEXT_X6);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[ 7]) == EXCEPTION_CONTEXT_X7);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[ 8]) == EXCEPTION_CONTEXT_X8);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[ 9]) == EXCEPTION_CONTEXT_X9);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[10]) == EXCEPTION_CONTEXT_X10);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[11]) == EXCEPTION_CONTEXT_X11);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[12]) == EXCEPTION_CONTEXT_X12);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[13]) == EXCEPTION_CONTEXT_X13);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[14]) == EXCEPTION_CONTEXT_X14);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[15]) == EXCEPTION_CONTEXT_X15);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[16]) == EXCEPTION_CONTEXT_X16);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[17]) == EXCEPTION_CONTEXT_X17);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[18]) == EXCEPTION_CONTEXT_X18);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[19]) == EXCEPTION_CONTEXT_X19);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[20]) == EXCEPTION_CONTEXT_X20);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[21]) == EXCEPTION_CONTEXT_X21);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[22]) == EXCEPTION_CONTEXT_X22);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[23]) == EXCEPTION_CONTEXT_X23);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[24]) == EXCEPTION_CONTEXT_X24);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[25]) == EXCEPTION_CONTEXT_X25);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[26]) == EXCEPTION_CONTEXT_X26);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[27]) == EXCEPTION_CONTEXT_X27);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[28]) == EXCEPTION_CONTEXT_X28);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[29]) == EXCEPTION_CONTEXT_X29);
+    static_assert(AMS_OFFSETOF(KExceptionContext, x[30]) == EXCEPTION_CONTEXT_X30);
+    static_assert(AMS_OFFSETOF(KExceptionContext, sp)    == EXCEPTION_CONTEXT_SP);
+    static_assert(AMS_OFFSETOF(KExceptionContext, pc)    == EXCEPTION_CONTEXT_PC);
+    static_assert(AMS_OFFSETOF(KExceptionContext, psr)   == EXCEPTION_CONTEXT_PSR);
+    static_assert(AMS_OFFSETOF(KExceptionContext, tpidr) == EXCEPTION_CONTEXT_TPIDR);
 
 }

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_k_interrupt_manager.hpp

@@ -105,35 +105,48 @@ namespace ams::kern::arch::arm64 {
             Result UnbindLocal(s32 irq);
             Result ClearGlobal(s32 irq);
             Result ClearLocal(s32 irq);
-        public:
-            static ALWAYS_INLINE u32 DisableInterrupts() {
+        private:
+            [[nodiscard]] static ALWAYS_INLINE u32 GetInterruptsEnabledState() {
                 u64 intr_state;
-                __asm__ __volatile__("mrs %[intr_state], daif\n"
-                                     "msr daifset, #2"
+                __asm__ __volatile__("mrs   %[intr_state], daif\n"
+                                     "ubfx  %[intr_state], %[intr_state], #7, #1"
                                      : [intr_state]"=r"(intr_state)
                                      :: "memory");
                 return intr_state;
             }
+        public:
+            static ALWAYS_INLINE void EnableInterrupts() {
+                __asm__ __volatile__("msr   daifclr, #2" ::: "memory");
+            }
 
-            static ALWAYS_INLINE u32 EnableInterrupts() {
-                u64 intr_state;
-                __asm__ __volatile__("mrs %[intr_state], daif\n"
-                                     "msr daifclr, #2"
-                                     : [intr_state]"=r"(intr_state)
-                                     :: "memory");
+            static ALWAYS_INLINE void DisableInterrupts() {
+                __asm__ __volatile__("msr   daifset, #2" ::: "memory");
+            }
+
+            [[nodiscard]] static ALWAYS_INLINE u32 GetInterruptsEnabledStateAndDisableInterrupts() {
+                const auto intr_state = GetInterruptsEnabledState();
+                DisableInterrupts();
+                return intr_state;
+            }
+
+            [[nodiscard]] static ALWAYS_INLINE u32 GetInterruptsEnabledStateAndEnableInterrupts() {
+                const auto intr_state = GetInterruptsEnabledState();
+                EnableInterrupts();
                 return intr_state;
             }
 
             static ALWAYS_INLINE void RestoreInterrupts(u32 intr_state) {
-                u64 cur_state;
-                __asm__ __volatile__("mrs %[cur_state], daif" : [cur_state]"=r"(cur_state));
-                __asm__ __volatile__("msr daif, %[intr_state]" :: [intr_state]"r"((cur_state & ~0x80ul) | (intr_state & 0x80)));
+                u64 tmp;
+                __asm__ __volatile__("mrs   %[tmp], daif\n"
+                                     "bfi   %[tmp], %[intr_state], #7, #1\n"
+                                     "msr daif, %[tmp]"
+                                     : [tmp]"=&r"(tmp)
+                                     : [intr_state]"r"(intr_state)
+                                     : "memory");
             }
 
             static ALWAYS_INLINE bool AreInterruptsEnabled() {
-                u64 intr_state;
-                __asm__ __volatile__("mrs %[intr_state], daif" : [intr_state]"=r"(intr_state));
-                return (intr_state & 0x80) == 0;
+                return GetInterruptsEnabledState() == 0;
             }
     };
 

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_k_interrupt_name.hpp

@@ -36,6 +36,10 @@ namespace ams::kern::arch::arm64 {
             KInterruptName_SecurePhysicalTimer    = 29,
             KInterruptName_NonSecurePhysicalTimer = 30,
             KInterruptName_LegacyNIrq             = 31,
+    #elif defined(ATMOSPHERE_BOARD_QEMU_VIRT)
+            KInterruptName_VirtualTimer           = 27,
+            KInterruptName_SecurePhysicalTimer    = 29,
+            KInterruptName_NonSecurePhysicalTimer = 30,
     #endif
 
     #if defined(ATMOSPHERE_BOARD_NINTENDO_NX)

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_k_page_table.hpp

@@ -168,7 +168,8 @@ namespace ams::kern::arch::arm64 {
                 return entry;
             }
         public:
-            constexpr KPageTable() : KPageTableBase(), m_manager(), m_ttbr(), m_asid() { /* ... */ }
+            constexpr explicit KPageTable(util::ConstantInitializeTag) : KPageTableBase(util::ConstantInitialize), m_manager(), m_ttbr(), m_asid() { /* ... */ }
+            explicit KPageTable() { /* ... */ }
 
             static NOINLINE void Initialize(s32 core_id);
 
@@ -218,27 +219,27 @@ namespace ams::kern::arch::arm64 {
 
             Result ChangePermissions(KProcessAddress virt_addr, size_t num_pages, PageTableEntry entry_template, DisableMergeAttribute disable_merge_attr, bool refresh_mapping, PageLinkedList *page_list, bool reuse_ll);
 
-            static void PteDataSynchronizationBarrier() {
+            static ALWAYS_INLINE void PteDataSynchronizationBarrier() {
                 cpu::DataSynchronizationBarrierInnerShareable();
             }
 
-            static void ClearPageTable(KVirtualAddress table) {
+            static ALWAYS_INLINE void ClearPageTable(KVirtualAddress table) {
                 cpu::ClearPageToZero(GetVoidPointer(table));
             }
 
-            void OnTableUpdated() const {
+            ALWAYS_INLINE void OnTableUpdated() const {
                 cpu::InvalidateTlbByAsid(m_asid);
             }
 
-            void OnKernelTableUpdated() const {
+            ALWAYS_INLINE void OnKernelTableUpdated() const {
                 cpu::InvalidateEntireTlbDataOnly();
             }
 
-            void OnKernelTableSinglePageUpdated(KProcessAddress virt_addr) const {
+            ALWAYS_INLINE void OnKernelTableSinglePageUpdated(KProcessAddress virt_addr) const {
                 cpu::InvalidateTlbByVaDataOnly(virt_addr);
             }
 
-            void NoteUpdated() const {
+            ALWAYS_INLINE void NoteUpdated() const {
                 cpu::DataSynchronizationBarrier();
 
                 if (this->IsKernel()) {
@@ -248,7 +249,7 @@ namespace ams::kern::arch::arm64 {
                 }
             }
 
-            void NoteSingleKernelPageUpdated(KProcessAddress virt_addr) const {
+            ALWAYS_INLINE void NoteSingleKernelPageUpdated(KProcessAddress virt_addr) const {
                 MESOSPHERE_ASSERT(this->IsKernel());
 
                 cpu::DataSynchronizationBarrier();

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_k_page_table_impl.hpp

@@ -42,7 +42,7 @@ namespace ams::kern::arch::arm64 {
                 const L3PageTableEntry *l3_entry;
             };
         private:
-            static constexpr size_t PageBits  = __builtin_ctzll(PageSize);
+            static constexpr size_t PageBits  = util::CountTrailingZeros(PageSize);
             static constexpr size_t NumLevels = 3;
             static constexpr size_t LevelBits = 9;
             static_assert(NumLevels > 0);
@@ -105,7 +105,9 @@ namespace ams::kern::arch::arm64 {
                 return GetL3EntryFromTable(KMemoryLayout::GetLinearVirtualAddress(entry->GetTable()), address);
             }
         public:
-            constexpr KPageTableImpl() : m_table(), m_is_kernel(), m_num_entries() { /* ... */ }
+            constexpr explicit KPageTableImpl(util::ConstantInitializeTag) : m_table(), m_is_kernel(), m_num_entries() { /* ... */ }
+
+            explicit KPageTableImpl() { /* ... */ }
 
             NOINLINE void InitializeForKernel(void *tb, KVirtualAddress start, KVirtualAddress end);
             NOINLINE void InitializeForProcess(void *tb, KVirtualAddress start, KVirtualAddress end);

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_k_process_page_table.hpp

@@ -23,8 +23,6 @@ namespace ams::kern::arch::arm64 {
         private:
             KPageTable m_page_table;
         public:
-            constexpr KProcessPageTable() : m_page_table() { /* ... */ }
-
             void Activate(u64 id) {
                 /* Activate the page table with the specified contextidr. */
                 m_page_table.Activate(id);

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_k_slab_heap_impl.hpp

@@ -45,6 +45,7 @@ namespace ams::kern::arch::arm64 {
 
             /* Select L1 cache. */
             cpu::SetCsselrEl1(0);
+            cpu::InstructionMemoryBarrier();
 
             /* Check that the L1 cache is not direct-mapped. */
             return cpu::CacheSizeIdRegisterAccessor().GetAssociativity() != 0;

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_k_supervisor_page_table.hpp

@@ -25,7 +25,7 @@ namespace ams::kern::arch::arm64 {
             KPageTable m_page_table;
             u64 m_ttbr0_identity[cpu::NumCores];
         public:
-            constexpr KSupervisorPageTable() : m_page_table(), m_ttbr0_identity() { /* ... */ }
+            constexpr KSupervisorPageTable() : m_page_table(util::ConstantInitialize), m_ttbr0_identity() { /* ... */ }
 
             NOINLINE void Initialize(s32 core_id);
 

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_k_thread_context.hpp

@@ -57,7 +57,8 @@ namespace ams::kern::arch::arm64 {
             static void RestoreFpuRegisters64(const KThreadContext &);
             static void RestoreFpuRegisters32(const KThreadContext &);
         public:
-            constexpr explicit KThreadContext() : m_callee_saved(), m_lr(), m_sp(), m_cpacr(), m_fpcr(), m_fpsr(), m_fpu_registers(), m_locked() { /* ... */ }
+            constexpr explicit KThreadContext(util::ConstantInitializeTag) : m_callee_saved(), m_lr(), m_sp(), m_cpacr(), m_fpcr(), m_fpsr(), m_fpu_registers(), m_locked() { /* ... */ }
+            explicit KThreadContext() { /* ... */ }
 
             Result Initialize(KVirtualAddress u_pc, KVirtualAddress k_sp, KVirtualAddress u_sp, uintptr_t arg, bool is_user, bool is_64_bit, bool is_main);
             Result Finalize();
@@ -86,25 +87,25 @@ namespace ams::kern::arch::arm64 {
     consteval bool KThreadContext::ValidateOffsets() {
         static_assert(sizeof(KThreadContext) == THREAD_CONTEXT_SIZE);
 
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.registers) == THREAD_CONTEXT_CPU_REGISTERS);
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.x19)       == THREAD_CONTEXT_X19);
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.x20)       == THREAD_CONTEXT_X20);
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.x21)       == THREAD_CONTEXT_X21);
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.x22)       == THREAD_CONTEXT_X22);
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.x23)       == THREAD_CONTEXT_X23);
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.x24)       == THREAD_CONTEXT_X24);
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.x25)       == THREAD_CONTEXT_X25);
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.x26)       == THREAD_CONTEXT_X26);
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.x27)       == THREAD_CONTEXT_X27);
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.x28)       == THREAD_CONTEXT_X28);
-        static_assert(__builtin_offsetof(KThreadContext, m_callee_saved.x29)       == THREAD_CONTEXT_X29);
-        static_assert(__builtin_offsetof(KThreadContext, m_lr)                     == THREAD_CONTEXT_LR);
-        static_assert(__builtin_offsetof(KThreadContext, m_sp)                     == THREAD_CONTEXT_SP);
-        static_assert(__builtin_offsetof(KThreadContext, m_cpacr)                  == THREAD_CONTEXT_CPACR);
-        static_assert(__builtin_offsetof(KThreadContext, m_fpcr)                   == THREAD_CONTEXT_FPCR);
-        static_assert(__builtin_offsetof(KThreadContext, m_fpsr)                   == THREAD_CONTEXT_FPSR);
-        static_assert(__builtin_offsetof(KThreadContext, m_fpu_registers)          == THREAD_CONTEXT_FPU_REGISTERS);
-        static_assert(__builtin_offsetof(KThreadContext, m_locked)                 == THREAD_CONTEXT_LOCKED);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.registers) == THREAD_CONTEXT_CPU_REGISTERS);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.x19)       == THREAD_CONTEXT_X19);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.x20)       == THREAD_CONTEXT_X20);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.x21)       == THREAD_CONTEXT_X21);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.x22)       == THREAD_CONTEXT_X22);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.x23)       == THREAD_CONTEXT_X23);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.x24)       == THREAD_CONTEXT_X24);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.x25)       == THREAD_CONTEXT_X25);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.x26)       == THREAD_CONTEXT_X26);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.x27)       == THREAD_CONTEXT_X27);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.x28)       == THREAD_CONTEXT_X28);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_callee_saved.x29)       == THREAD_CONTEXT_X29);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_lr)                     == THREAD_CONTEXT_LR);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_sp)                     == THREAD_CONTEXT_SP);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_cpacr)                  == THREAD_CONTEXT_CPACR);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_fpcr)                   == THREAD_CONTEXT_FPCR);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_fpsr)                   == THREAD_CONTEXT_FPSR);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_fpu_registers)          == THREAD_CONTEXT_FPU_REGISTERS);
+        static_assert(AMS_OFFSETOF(KThreadContext, m_locked)                 == THREAD_CONTEXT_LOCKED);
 
         return true;
     }

libraries/libmesosphere/include/mesosphere/arch/arm64/kern_secure_monitor_base.hpp

@@ -0,0 +1,95 @@
+/*
+ * Copyright (c) Atmosphère-NX
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#pragma once
+#include <mesosphere/kern_common.hpp>
+#include <mesosphere/kern_select_cpu.hpp>
+#include <mesosphere/kern_select_interrupt_manager.hpp>
+
+namespace ams::kern::arch::arm64::smc {
+
+    template<int SmcId, bool DisableInterrupt>
+    void SecureMonitorCall(u64 *buf) {
+        /* Load arguments into registers. */
+        register u64 x0 asm("x0") = buf[0];
+        register u64 x1 asm("x1") = buf[1];
+        register u64 x2 asm("x2") = buf[2];
+        register u64 x3 asm("x3") = buf[3];
+        register u64 x4 asm("x4") = buf[4];
+        register u64 x5 asm("x5") = buf[5];
+        register u64 x6 asm("x6") = buf[6];
+        register u64 x7 asm("x7") = buf[7];
+
+        /* Perform the call. */
+        if constexpr (DisableInterrupt) {
+            KScopedInterruptDisable di;
+
+            /* Backup the current thread pointer. */
+            const uintptr_t current_thread_pointer_value = cpu::GetCurrentThreadPointerValue();
+
+            __asm__ __volatile__("smc %c[smc_id]"
+                                : "+r"(x0), "+r"(x1), "+r"(x2), "+r"(x3), "+r"(x4), "+r"(x5), "+r"(x6), "+r"(x7)
+                                : [smc_id]"i"(SmcId)
+                                : "x8", "x9", "x10", "x11", "x12", "x13", "x14", "x15", "x16", "x17", "x18", "cc", "memory"
+                                );
+
+            /* Restore the current thread pointer into X18. */
+            cpu::SetCurrentThreadPointerValue(current_thread_pointer_value);
+        } else {
+            /* Backup the current thread pointer. */
+            const uintptr_t current_thread_pointer_value = cpu::GetCurrentThreadPointerValue();
+
+            __asm__ __volatile__("smc %c[smc_id]"
+                                : "+r"(x0), "+r"(x1), "+r"(x2), "+r"(x3), "+r"(x4), "+r"(x5), "+r"(x6), "+r"(x7)
+                                : [smc_id]"i"(SmcId)
+                                : "x8", "x9", "x10", "x11", "x12", "x13", "x14", "x15", "x16", "x17", "x18", "cc", "memory"
+                                );
+
+            /* Restore the current thread pointer into X18. */
+            cpu::SetCurrentThreadPointerValue(current_thread_pointer_value);
+        }
+
+        /* Store arguments to output. */
+        buf[0] = x0;
+        buf[1] = x1;
+        buf[2] = x2;
+        buf[3] = x3;
+        buf[4] = x4;
+        buf[5] = x5;
+        buf[6] = x6;
+        buf[7] = x7;
+    }
+
+    enum PsciFunction {
+        PsciFunction_CpuSuspend = 0xC4000001,
+        PsciFunction_CpuOff     = 0x84000002,
+        PsciFunction_CpuOn      = 0xC4000003,
+    };
+
+    template<int SmcId, bool DisableInterrupt>
+    u64 PsciCall(PsciFunction function, u64 x1 = 0, u64 x2 = 0, u64 x3 = 0, u64 x4 = 0, u64 x5 = 0, u64 x6 = 0, u64 x7 = 0) {
+        ams::svc::lp64::SecureMonitorArguments args = { { function, x1, x2, x3, x4, x5, x6, x7 } };
+
+        SecureMonitorCall<SmcId, DisableInterrupt>(args.r);
+
+        return args.r[0];
+    }
+
+    template<int SmcId, bool DisableInterrupt>
+    u64 CpuOn(u64 core_id, uintptr_t entrypoint, uintptr_t arg) {
+        return PsciCall<SmcId, DisableInterrupt>(PsciFunction_CpuOn, core_id, entrypoint, arg);
+    }
+
+}

libraries/libmesosphere/include/mesosphere/board/generic/kern_k_device_page_table.hpp

@@ -44,13 +44,13 @@ namespace ams::kern::board::generic {
                 return ams::kern::svc::ResultNotImplemented();
             }
 
-            Result ALWAYS_INLINE Map(size_t *out_mapped_size, const KPageGroup &pg, KDeviceVirtualAddress device_address, ams::svc::MemoryPermission device_perm, bool refresh_mappings) {
-                MESOSPHERE_UNUSED(out_mapped_size, pg, device_address, device_perm, refresh_mappings);
+            Result ALWAYS_INLINE Map(KProcessPageTable *page_table, KProcessAddress process_address, size_t size, KDeviceVirtualAddress device_address, ams::svc::MemoryPermission device_perm, bool is_aligned) {
+                MESOSPHERE_UNUSED(page_table, process_address, size, device_address, device_perm, is_aligned);
                 return ams::kern::svc::ResultNotImplemented();
             }
 
-            Result ALWAYS_INLINE Unmap(const KPageGroup &pg, KDeviceVirtualAddress device_address) {
-                MESOSPHERE_UNUSED(pg, device_address);
+            Result ALWAYS_INLINE Unmap(KProcessPageTable *page_table, KProcessAddress process_address, size_t size, KDeviceVirtualAddress device_address) {
+                MESOSPHERE_UNUSED(page_table, process_address, size, device_address);
                 return ams::kern::svc::ResultNotImplemented();
             }
 

libraries/libmesosphere/include/mesosphere/board/nintendo/nx/kern_k_device_page_table.hpp

@@ -61,7 +61,12 @@ namespace ams::kern::board::nintendo::nx {
                 return KPageTable::GetPageTablePhysicalAddress(addr);
             }
         public:
-            constexpr KDevicePageTable() : m_tables(), m_table_asids(), m_attached_device(), m_attached_value(), m_detached_value(), m_hs_attached_value(), m_hs_detached_value() { /* ... */ }
+            constexpr KDevicePageTable()
+                : m_tables{Null<KVirtualAddress>, Null<KVirtualAddress>, Null<KVirtualAddress>, Null<KVirtualAddress>},
+                  m_table_asids(), m_attached_device(), m_attached_value(), m_detached_value(), m_hs_attached_value(), m_hs_detached_value()
+            {
+                /* ... */
+            }
 
             Result Initialize(u64 space_address, u64 space_size);
             void Finalize();

libraries/libmesosphere/include/mesosphere/board/nintendo/nx/kern_k_memory_layout.hpp

@@ -15,9 +15,12 @@
  */
 #pragma once
 #include <mesosphere/kern_common.hpp>
+#include <mesosphere/kern_k_typed_address.hpp>
 
 namespace ams::kern {
 
+    constexpr inline KPhysicalAddress MainMemoryAddress = 0x80000000;
+
     constexpr inline size_t MainMemorySize    = 4_GB;
     constexpr inline size_t MainMemorySizeMax = 8_GB;
 

libraries/libmesosphere/include/mesosphere/board/nintendo/nx/kern_k_system_control.hpp

@@ -15,23 +15,17 @@
  */
 #pragma once
 #include <mesosphere/kern_common.hpp>
-
-namespace ams::kern {
-
-    struct InitialProcessBinaryLayout;
-
-}
+#include <mesosphere/kern_k_system_control_base.hpp>
 
 namespace ams::kern::board::nintendo::nx {
 
-    class KSystemControl {
+    class KSystemControl : public KSystemControlBase {
         public:
-            class Init {
+            class Init : public KSystemControlBase::Init {
                 public:
                     /* Initialization. */
+                    static size_t GetRealMemorySize();
                     static size_t GetIntendedMemorySize();
-                    static KPhysicalAddress GetKernelPhysicalBaseAddress(uintptr_t base_address);
-                    static void GetInitialProcessBinaryLayout(InitialProcessBinaryLayout *out);
                     static bool ShouldIncreaseThreadResourceLimit();
                     static void CpuOn(u64 core_id, uintptr_t entrypoint, uintptr_t arg);
                     static size_t GetApplicationPoolSize();
@@ -40,7 +34,7 @@ namespace ams::kern::board::nintendo::nx {
                     static u8 GetDebugLogUartPort();
 
                     /* Randomness. */
-                    static void GenerateRandomBytes(void *dst, size_t size);
+                    static void GenerateRandom(u64 *dst, size_t count);
                     static u64  GenerateRandomRange(u64 min, u64 max);
             };
         public:
@@ -50,7 +44,7 @@ namespace ams::kern::board::nintendo::nx {
             static NOINLINE u32 GetCreateProcessMemoryPool();
 
             /* Randomness. */
-            static void GenerateRandomBytes(void *dst, size_t size);
+            static void GenerateRandom(u64 *dst, size_t count);
             static u64  GenerateRandomRange(u64 min, u64 max);
             static u64  GenerateRandomU64();
 
@@ -58,23 +52,12 @@ namespace ams::kern::board::nintendo::nx {
             static void ReadWriteRegisterPrivileged(u32 *out, ams::svc::PhysicalAddress address, u32 mask, u32 value);
             static Result ReadWriteRegister(u32 *out, ams::svc::PhysicalAddress address, u32 mask, u32 value);
 
-            static ALWAYS_INLINE u32 ReadRegisterPrivileged(ams::svc::PhysicalAddress address) {
-                u32 v;
-                ReadWriteRegisterPrivileged(std::addressof(v), address, 0x00000000u, 0);
-                return v;
-            }
-
-            static ALWAYS_INLINE void WriteRegisterPrivileged(ams::svc::PhysicalAddress address, u32 value) {
-                u32 v;
-                ReadWriteRegisterPrivileged(std::addressof(v), address, 0xFFFFFFFFu, value);
-            }
-
             /* Power management. */
             static void SleepSystem();
             static NORETURN void StopSystem(void *arg = nullptr);
 
             /* User access. */
-            static void CallSecureMonitorFromUser(ams::svc::lp64::SecureMonitorArguments *args);
+            static void CallSecureMonitorFromUserImpl(ams::svc::lp64::SecureMonitorArguments *args);
 
             /* Secure Memory. */
             static size_t CalculateRequiredSecureMemorySize(size_t size, u32 pool);

libraries/libmesosphere/include/mesosphere/board/qemu/virt/kern_cpu_map.hpp

@@ -0,0 +1,33 @@
+/*
+ * Copyright (c) Atmosphère-NX
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#pragma once
+#include <mesosphere/kern_common.hpp>
+
+namespace ams::kern::board::qemu::virt::impl::cpu {
+
+    /* Virtual to Physical core map. */
+    constexpr inline const s32 VirtualToPhysicalCoreMap[BITSIZEOF(u64)] = {
+        0, 1, 2, 3, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 3,
+    };
+
+}

libraries/libmesosphere/include/mesosphere/board/qemu/virt/kern_k_memory_layout.hpp

@@ -0,0 +1,27 @@
+/*
+ * Copyright (c) Atmosphère-NX
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#pragma once
+#include <mesosphere/kern_common.hpp>
+#include <mesosphere/kern_k_typed_address.hpp>
+
+namespace ams::kern {
+
+    constexpr inline KPhysicalAddress MainMemoryAddress = 0x40000000;
+
+    constexpr inline size_t MainMemorySize    = 4_GB;
+    constexpr inline size_t MainMemorySizeMax = 8_GB;
+
+}

libraries/libmesosphere/include/mesosphere/board/qemu/virt/kern_k_memory_region_device_types.inc

@@ -0,0 +1,20 @@
+/*
+ * Copyright (c) Atmosphère-NX
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+/* All architectures must define NumBoardDeviceRegions. */
+constexpr inline const auto NumBoardDeviceRegions = 0;
+                                                                                                /* UNUSED: .Derive(NumBoardDeviceRegions, 0); */
+

libraries/libmesosphere/include/mesosphere/board/qemu/virt/kern_k_system_control.hpp

@@ -0,0 +1,28 @@
+/*
+ * Copyright (c) Atmosphère-NX
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#pragma once
+#include <mesosphere/kern_common.hpp>
+#include <mesosphere/kern_k_system_control_base.hpp>
+
+namespace ams::kern::board::qemu::virt {
+
+    class KSystemControl : public KSystemControlBase {
+        public:
+            /* User access. */
+            static void CallSecureMonitorFromUser(ams::svc::lp64::SecureMonitorArguments *args);
+    };
+
+}

libraries/libmesosphere/include/mesosphere/kern_build_config.hpp

@@ -32,3 +32,25 @@
 //#define MESOSPHERE_BUILD_FOR_TRACING
 #define MESOSPHERE_ENABLE_PANIC_REGISTER_DUMP
 #define MESOSPHERE_ENABLE_HARDWARE_SINGLE_STEP
+
+/* NOTE: This enables fast class token storage using a class member. */
+/* This saves a virtual call when doing KAutoObject->DynCast<>(), */
+/* at the cost of storing class tokens inside the class object. */
+/* However, as of (10/16/2021) KAutoObject has an unused class member */
+/* of the right side, and so this does not actually cost any space. */
+#define MESOSPHERE_ENABLE_DEVIRTUALIZED_DYNAMIC_CAST
+
+/* NOTE: This enables usage of KDebug handles as parameter for svc::GetInfo */
+/* calls which require a process parameter. This enables a debugger to obtain */
+/* address space/layout information, for example. However, it changes abi, and so */
+/* this define allows toggling the extension. */
+#define MESOSPHERE_ENABLE_GET_INFO_OF_DEBUG_PROCESS
+
+/* NOTE: This uses currently-reserved bits inside the MapRange capability */
+/* in order to support large physical addresses (40-bit instead of 36). */
+/* This is toggleable in order to disable it if N ever uses those bits. */
+#if defined(ATMOSPHERE_BOARD_NINTENDO_NX)
+//#define MESOSPHERE_ENABLE_LARGE_PHYSICAL_ADDRESS_CAPABILITIES
+#else
+#define MESOSPHERE_ENABLE_LARGE_PHYSICAL_ADDRESS_CAPABILITIES
+#endif

libraries/libmesosphere/include/mesosphere/kern_debug_log.hpp

@@ -43,6 +43,8 @@ namespace ams::kern {
 
     #ifdef ATMOSPHERE_BOARD_NINTENDO_NX
         #define MESOSPHERE_DEBUG_LOG_USE_UART
+    #elif defined(ATMOSPHERE_BOARD_QEMU_VIRT)
+        #define MESOSPHERE_DEBUG_LOG_USE_SEMIHOSTING
     #else
         #error "Unknown board for Default Debug Log Source"
     #endif

libraries/libmesosphere/include/mesosphere/kern_k_address_arbiter.hpp

@@ -25,7 +25,7 @@ namespace ams::kern {
         private:
             ThreadTree m_tree;
         public:
-            constexpr KAddressArbiter() : m_tree() { /* ... */ }
+            constexpr KAddressArbiter() = default;
 
             Result SignalToAddress(uintptr_t addr, ams::svc::SignalType type, s32 value, s32 count) {
                 switch (type) {

libraries/libmesosphere/include/mesosphere/kern_k_auto_object.hpp

@@ -22,27 +22,73 @@ namespace ams::kern {
 
     class KProcess;
 
+    #if defined(MESOSPHERE_BUILD_FOR_DEBUGGING) || defined(MESOSPHERE_BUILD_FOR_AUDITING)
+    #define MESOSPHERE_AUTO_OBJECT_TYPENAME_IMPL(CLASS) #CLASS
+    #else
+    #define MESOSPHERE_AUTO_OBJECT_TYPENAME_IMPL(CLASS) ""
+    #endif
+
     #define MESOSPHERE_AUTOOBJECT_TRAITS(CLASS, BASE_CLASS)                                                     \
         NON_COPYABLE(CLASS);                                                                                    \
         NON_MOVEABLE(CLASS);                                                                                    \
         private:                                                                                                \
             friend class ::ams::kern::KClassTokenGenerator;                                                     \
             static constexpr inline auto ObjectType = ::ams::kern::KClassTokenGenerator::ObjectType::CLASS;     \
-            static constexpr inline const char * const TypeName = #CLASS;                                       \
+            static constexpr inline const char * const TypeName = MESOSPHERE_AUTO_OBJECT_TYPENAME_IMPL(CLASS);  \
             static constexpr inline ClassTokenType ClassToken() { return ::ams::kern::ClassToken<CLASS>; }      \
         public:                                                                                                 \
             using BaseClass = BASE_CLASS;                                                                       \
-            static constexpr ALWAYS_INLINE TypeObj GetStaticTypeObj() {                                         \
+            static consteval ALWAYS_INLINE TypeObj GetStaticTypeObj() {                                         \
                 constexpr ClassTokenType Token = ClassToken();                                                  \
                 return TypeObj(TypeName, Token);                                                                \
             }                                                                                                   \
-            static constexpr ALWAYS_INLINE const char *GetStaticTypeName() { return TypeName; }                 \
+            static consteval ALWAYS_INLINE const char *GetStaticTypeName() { return TypeName; }                 \
             virtual TypeObj GetTypeObj() const { return GetStaticTypeObj(); }                                   \
             virtual const char *GetTypeName() { return GetStaticTypeName(); }                                   \
         private:
 
-
     class KAutoObject {
+        public:
+            class ReferenceCount {
+                NON_COPYABLE(ReferenceCount);
+                NON_MOVEABLE(ReferenceCount);
+                private:
+                    using Storage = u32;
+                private:
+                    util::Atomic<Storage> m_value;
+                public:
+                    ALWAYS_INLINE explicit ReferenceCount() { /* ... */ }
+                    constexpr ALWAYS_INLINE explicit ReferenceCount(Storage v) : m_value(v) { /* ... */ }
+
+                    ALWAYS_INLINE void operator=(Storage v) { m_value = v; }
+
+                    ALWAYS_INLINE Storage GetValue() const { return m_value.Load(); }
+
+                    ALWAYS_INLINE bool Open() {
+                        /* Atomically increment the reference count, only if it's positive. */
+                        u32 cur = m_value.Load<std::memory_order_relaxed>();
+                        do {
+                            if (AMS_UNLIKELY(cur == 0)) {
+                                MESOSPHERE_AUDIT(cur != 0);
+                                return false;
+                            }
+                            MESOSPHERE_ABORT_UNLESS(cur < cur + 1);
+                        } while (AMS_UNLIKELY(!m_value.CompareExchangeWeak<std::memory_order_relaxed>(cur, cur + 1)));
+
+                        return true;
+                    }
+
+                    ALWAYS_INLINE bool Close() {
+                        /* Atomically decrement the reference count, not allowing it to become negative. */
+                        u32 cur = m_value.Load<std::memory_order_relaxed>();
+                        do {
+                            MESOSPHERE_ABORT_UNLESS(cur > 0);
+                        } while (AMS_UNLIKELY(!m_value.CompareExchangeWeak<std::memory_order_relaxed>(cur, cur - 1)));
+
+                        /* Return whether the object was closed. */
+                        return cur - 1 == 0;
+                    }
+            };
         protected:
             class TypeObj {
                 private:
@@ -63,37 +109,61 @@ namespace ams::kern {
                     }
 
                     constexpr ALWAYS_INLINE bool IsDerivedFrom(const TypeObj &rhs) {
-                        return (this->GetClassToken() | rhs.GetClassToken()) == this->GetClassToken();
+                        return IsClassTokenDerivedFrom(this->GetClassToken(), rhs.GetClassToken());
+                    }
+
+                    static constexpr ALWAYS_INLINE bool IsClassTokenDerivedFrom(ClassTokenType derived, ClassTokenType base) {
+                        return (derived | base) == derived;
                     }
             };
         private:
             MESOSPHERE_AUTOOBJECT_TRAITS(KAutoObject, KAutoObject);
         private:
             KAutoObject *m_next_closed_object;
-            std::atomic<u32> m_ref_count;
+            ReferenceCount m_ref_count;
+            #if defined(MESOSPHERE_ENABLE_DEVIRTUALIZED_DYNAMIC_CAST)
+            ClassTokenType m_class_token;
+            #endif
         public:
-            static KAutoObject *Create(KAutoObject *ptr);
-        public:
-            constexpr ALWAYS_INLINE explicit KAutoObject() : m_next_closed_object(nullptr), m_ref_count(0) { MESOSPHERE_ASSERT_THIS(); }
+            constexpr ALWAYS_INLINE explicit KAutoObject(util::ConstantInitializeTag) : m_next_closed_object(nullptr), m_ref_count(0)
+            #if defined(MESOSPHERE_ENABLE_DEVIRTUALIZED_DYNAMIC_CAST)
+                , m_class_token(0)
+            #endif
+            {
+                MESOSPHERE_ASSERT_THIS();
+            }
+
+            ALWAYS_INLINE explicit KAutoObject() : m_ref_count(0) { MESOSPHERE_ASSERT_THIS(); }
 
             /* Destroy is responsible for destroying the auto object's resources when ref_count hits zero. */
             virtual void Destroy() { MESOSPHERE_ASSERT_THIS(); }
 
             /* Finalize is responsible for cleaning up resource, but does not destroy the object. */
-            virtual void Finalize() { MESOSPHERE_ASSERT_THIS(); }
+            /* NOTE: This is a virtual function in official kernel, but because everything which uses it */
+            /* is already using CRTP for slab heap, we have devirtualized it for performance gain. */
+            /* virtual void Finalize() { MESOSPHERE_ASSERT_THIS(); } */
 
-            virtual KProcess *GetOwner() const { return nullptr; }
+            /* NOTE: This is a virtual function which is unused-except-for-debug in Nintendo's kernel. */
+            /* virtual KProcess *GetOwner() const { return nullptr; } */
 
             u32 GetReferenceCount() const {
-                return m_ref_count.load();
+                return m_ref_count.GetValue();
             }
 
             ALWAYS_INLINE bool IsDerivedFrom(const TypeObj &rhs) const {
-                return this->GetTypeObj().IsDerivedFrom(rhs);
+                #if defined(MESOSPHERE_ENABLE_DEVIRTUALIZED_DYNAMIC_CAST)
+                    return TypeObj::IsClassTokenDerivedFrom(m_class_token, rhs.GetClassToken());
+                #else
+                    return this->GetTypeObj().IsDerivedFrom(rhs);
+                #endif
             }
 
             ALWAYS_INLINE bool IsDerivedFrom(const KAutoObject &rhs) const {
-                return this->IsDerivedFrom(rhs.GetTypeObj());
+                #if defined(MESOSPHERE_ENABLE_DEVIRTUALIZED_DYNAMIC_CAST)
+                    return TypeObj::IsClassTokenDerivedFrom(m_class_token, rhs.m_class_token);
+                #else
+                    return this->IsDerivedFrom(rhs.GetTypeObj());
+                #endif
             }
 
             template<typename Derived>
@@ -120,8 +190,19 @@ namespace ams::kern {
                 }
             }
 
-            bool Open();
-            void Close();
+            NOINLINE bool Open() {
+                MESOSPHERE_ASSERT_THIS();
+
+                return m_ref_count.Open();
+            }
+
+            NOINLINE void Close() {
+                MESOSPHERE_ASSERT_THIS();
+
+                if (m_ref_count.Close()) {
+                    this->ScheduleDestruction();
+                }
+            }
         private:
             /* NOTE: This has to be defined *after* KThread is defined. */
             /* Nintendo seems to handle this by defining Open/Close() in a cpp, but we'd like them to remain in headers. */
@@ -130,34 +211,46 @@ namespace ams::kern {
         public:
             /* Getter, for KThread. */
             ALWAYS_INLINE KAutoObject *GetNextClosedObject() { return m_next_closed_object; }
+        public:
+            template<typename Derived> requires (std::derived_from<Derived, KAutoObject>)
+            static ALWAYS_INLINE void Create(typename std::type_identity<Derived>::type *obj) {
+                /* Get auto object pointer. */
+                KAutoObject &auto_object = *static_cast<KAutoObject *>(obj);
+
+                /* If we should, set our class token. */
+                #if defined(MESOSPHERE_ENABLE_DEVIRTUALIZED_DYNAMIC_CAST)
+                {
+                    constexpr auto Token = Derived::GetStaticTypeObj().GetClassToken();
+                    auto_object.m_class_token = Token;
+                }
+                #endif
+
+                /* Initialize reference count to 1. */
+                auto_object.m_ref_count = 1;
+            }
     };
 
-    class KAutoObjectWithListContainer;
-
-    class KAutoObjectWithList : public KAutoObject {
+    class KAutoObjectWithListBase : public KAutoObject {
         private:
+            void *m_alignment_forcer_unused[0];
+        public:
+            constexpr ALWAYS_INLINE explicit KAutoObjectWithListBase(util::ConstantInitializeTag) : KAutoObject(util::ConstantInitialize), m_alignment_forcer_unused{} { /* ... */ }
+
+            ALWAYS_INLINE explicit KAutoObjectWithListBase() { /* ... */ }
+    };
+
+    class KAutoObjectWithList : public KAutoObjectWithListBase {
+        private:
+            template<typename>
             friend class KAutoObjectWithListContainer;
         private:
             util::IntrusiveRedBlackTreeNode m_list_node;
         public:
-            constexpr ALWAYS_INLINE KAutoObjectWithList() : m_list_node() { /* ... */ }
-
-            static ALWAYS_INLINE int Compare(const KAutoObjectWithList &lhs, const KAutoObjectWithList &rhs) {
-                const u64 lid = lhs.GetId();
-                const u64 rid = rhs.GetId();
-
-                if (lid < rid) {
-                    return -1;
-                } else if (lid > rid) {
-                    return 1;
-                } else {
-                    return 0;
-                }
-            }
+            constexpr ALWAYS_INLINE KAutoObjectWithList(util::ConstantInitializeTag) : KAutoObjectWithListBase(util::ConstantInitialize), m_list_node(util::ConstantInitialize) { /* ... */ }
+            ALWAYS_INLINE explicit KAutoObjectWithList() { /* ... */ }
         public:
-            virtual u64 GetId() const {
-                return reinterpret_cast<u64>(this);
-            }
+            /* NOTE: This is virtual in Nintendo's kernel. */
+            u64 GetId() const;
     };
 
     template<typename T> requires std::derived_from<T, KAutoObject>
@@ -173,7 +266,6 @@ namespace ams::kern {
                 std::swap(m_obj, rhs.m_obj);
             }
         public:
-            constexpr ALWAYS_INLINE KScopedAutoObject() : m_obj(nullptr) { /* ... */ }
             constexpr ALWAYS_INLINE KScopedAutoObject(T *o) : m_obj(o) {
                 if (m_obj != nullptr) {
                     m_obj->Open();
@@ -228,5 +320,52 @@ namespace ams::kern {
             constexpr ALWAYS_INLINE bool IsNotNull() const { return m_obj != nullptr; }
     };
 
+    template<typename T> requires std::derived_from<T, KAutoObject>
+    class KSharedAutoObject {
+        private:
+            T *m_object;
+            KAutoObject::ReferenceCount m_ref_count;
+        public:
+            explicit KSharedAutoObject() : m_object(nullptr) { /* ... */ }
+
+            void Attach(T *obj) {
+                MESOSPHERE_ASSERT(m_object == nullptr);
+
+                /* Set our object. */
+                m_object = obj;
+
+                /* Open reference to our object. */
+                m_object->Open();
+
+                /* Set our reference count. */
+                m_ref_count = 1;
+            }
+
+            bool Open() {
+                return m_ref_count.Open();
+            }
+
+            void Close() {
+                if (m_ref_count.Close()) {
+                    this->Detach();
+                }
+            }
+
+            ALWAYS_INLINE T *Get() const {
+                return m_object;
+            }
+        private:
+            void Detach() {
+                /* Close our object, if we have one. */
+                if (T * const object = m_object; AMS_LIKELY(object != nullptr)) {
+                    /* Set our object to a debug sentinel value, which will cause crash if accessed. */
+                    m_object = reinterpret_cast<T *>(1);
+
+                    /* Close reference to our object. */
+                    object->Close();
+                }
+            }
+    };
+
 
 }

libraries/libmesosphere/include/mesosphere/kern_k_auto_object_container.hpp

@@ -20,45 +20,131 @@
 
 namespace ams::kern {
 
-    class KAutoObjectWithListContainer {
-        NON_COPYABLE(KAutoObjectWithListContainer);
-        NON_MOVEABLE(KAutoObjectWithListContainer);
-        public:
-            using ListType = util::IntrusiveRedBlackTreeMemberTraits<&KAutoObjectWithList::m_list_node>::TreeType<KAutoObjectWithList>;
-        public:
-            class ListAccessor : public KScopedLightLock {
-                private:
-                    ListType &m_list;
-                public:
-                    explicit ListAccessor(KAutoObjectWithListContainer *container) : KScopedLightLock(container->m_lock), m_list(container->m_object_list) { /* ... */ }
-                    explicit ListAccessor(KAutoObjectWithListContainer &container) : KScopedLightLock(container.m_lock), m_list(container.m_object_list) { /* ... */ }
+    namespace impl {
 
-                    typename ListType::iterator begin() const {
-                        return m_list.begin();
+        template<typename T>
+        struct GetAutoObjectWithListComparator;
+
+        class KAutoObjectWithListContainerBase {
+            NON_COPYABLE(KAutoObjectWithListContainerBase);
+            NON_MOVEABLE(KAutoObjectWithListContainerBase);
+            protected:
+                template<typename ListType>
+                class ListAccessorImpl {
+                    NON_COPYABLE(ListAccessorImpl);
+                    NON_MOVEABLE(ListAccessorImpl);
+                    private:
+                        KScopedLightLock m_lk;
+                        ListType &m_list;
+                    public:
+                        explicit ALWAYS_INLINE ListAccessorImpl(KAutoObjectWithListContainerBase *container, ListType &list) : m_lk(container->m_lock), m_list(list) { /* ... */ }
+                        explicit ALWAYS_INLINE ListAccessorImpl(KAutoObjectWithListContainerBase &container, ListType &list) : m_lk(container.m_lock), m_list(list) { /* ... */ }
+
+                        ALWAYS_INLINE ~ListAccessorImpl() { /* ... */ }
+
+                        ALWAYS_INLINE typename ListType::iterator begin() const {
+                            return m_list.begin();
+                        }
+
+                        ALWAYS_INLINE typename ListType::iterator end() const {
+                            return m_list.end();
+                        }
+
+                        ALWAYS_INLINE typename ListType::iterator find(typename ListType::const_reference ref) const {
+                            return m_list.find(ref);
+                        }
+
+                        ALWAYS_INLINE typename ListType::iterator find_key(typename ListType::const_key_reference ref) const {
+                            return m_list.find_key(ref);
+                        }
+                };
+
+                template<typename ListType>
+                friend class ListAccessorImpl;
+            private:
+                KLightLock m_lock;
+            protected:
+                constexpr KAutoObjectWithListContainerBase() : m_lock() { /* ... */ }
+
+                ALWAYS_INLINE void InitializeImpl() { MESOSPHERE_ASSERT_THIS(); }
+                ALWAYS_INLINE void FinalizeImpl() { MESOSPHERE_ASSERT_THIS(); }
+
+                template<typename ListType>
+                void RegisterImpl(KAutoObjectWithList *obj, ListType &list) {
+                    MESOSPHERE_ASSERT_THIS();
+
+                    KScopedLightLock lk(m_lock);
+
+                    list.insert(*obj);
+                }
+
+                template<typename ListType>
+                void UnregisterImpl(KAutoObjectWithList *obj, ListType &list) {
+                    MESOSPHERE_ASSERT_THIS();
+
+                    KScopedLightLock lk(m_lock);
+
+                    list.erase(list.iterator_to(*obj));
+                }
+
+                template<typename U, typename ListType>
+                size_t GetOwnedCountImpl(const KProcess *owner, ListType &list) {
+                    MESOSPHERE_ASSERT_THIS();
+
+                    KScopedLightLock lk(m_lock);
+
+                    size_t count = 0;
+
+                    for (const auto &obj : list) {
+                        MESOSPHERE_AUDIT(obj.DynamicCast<const U *>() != nullptr);
+                        if (static_cast<const U &>(obj).GetOwner() == owner) {
+                            ++count;
+                        }
                     }
 
-                    typename ListType::iterator end() const {
-                        return m_list.end();
-                    }
+                    return count;
+                }
+        };
 
-                    typename ListType::iterator find(typename ListType::const_reference ref) const {
-                        return m_list.find(ref);
-                    }
-            };
+        struct DummyKAutoObjectWithListComparator {
+            static NOINLINE int Compare(KAutoObjectWithList &lhs, KAutoObjectWithList &rhs) {
+                MESOSPHERE_UNUSED(lhs, rhs);
+                MESOSPHERE_PANIC("DummyKAutoObjectWithListComparator invoked");
+            }
+        };
 
-            friend class ListAccessor;
+    }
+
+    template<typename T>
+    class KAutoObjectWithListContainer : public impl::KAutoObjectWithListContainerBase {
         private:
-            KLightLock m_lock;
-            ListType m_object_list;
+            using Base = impl::KAutoObjectWithListContainerBase;
         public:
-            constexpr KAutoObjectWithListContainer() : m_lock(), m_object_list() { MESOSPHERE_ASSERT_THIS(); }
+            class ListAccessor;
+            friend class ListAccessor;
 
-            void Initialize() { MESOSPHERE_ASSERT_THIS(); }
-            void Finalize() { MESOSPHERE_ASSERT_THIS(); }
+            template<typename Comparator>
+            using ListType = util::IntrusiveRedBlackTreeMemberTraits<&KAutoObjectWithList::m_list_node>::TreeType<Comparator>;
 
-            void Register(KAutoObjectWithList *obj);
-            void Unregister(KAutoObjectWithList *obj);
-            size_t GetOwnedCount(KProcess *owner);
+            using DummyListType = ListType<impl::DummyKAutoObjectWithListComparator>;
+        private:
+            DummyListType m_dummy_object_list;
+        public:
+            constexpr ALWAYS_INLINE KAutoObjectWithListContainer() : Base(), m_dummy_object_list() { static_assert(std::derived_from<T, KAutoObjectWithList>); }
+
+            ALWAYS_INLINE void Initialize() { return this->InitializeImpl(); }
+            ALWAYS_INLINE void Finalize() { return this->FinalizeImpl(); }
+
+            void Register(T *obj);
+            void Unregister(T *obj);
+
+        private:
+            size_t GetOwnedCountChecked(const KProcess *owner);
+        public:
+            template<typename U> requires (std::same_as<U, T> && requires (const U &u) { { u.GetOwner() } -> std::convertible_to<const KProcess *>; })
+            ALWAYS_INLINE size_t GetOwnedCount(const KProcess *owner) {
+                return this->GetOwnedCountChecked(owner);
+            }
     };
 
 

libraries/libmesosphere/include/mesosphere/kern_k_auto_object_impls.hpp

@@ -0,0 +1,163 @@
+/*
+ * Copyright (c) Atmosphère-NX
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#pragma once
+#include <mesosphere/kern_common.hpp>
+#include <mesosphere/kern_k_class_token.hpp>
+
+namespace ams::kern {
+
+    /* NOTE: This header is included after all other KAutoObjects. */
+    namespace impl {
+
+        template<typename T> requires std::derived_from<T, KAutoObject>
+        consteval bool IsAutoObjectInheritanceValidImpl() {
+            #define CLASS_TOKEN_HANDLER(CLASSNAME)                                                                               \
+                if constexpr (std::same_as<T, CLASSNAME>) {                                                                      \
+                    if (T::GetStaticTypeObj().GetClassToken() != ::ams::kern::ClassToken<CLASSNAME>) {                           \
+                        return false;                                                                                            \
+                    }                                                                                                            \
+                } else {                                                                                                         \
+                    if (T::GetStaticTypeObj().IsDerivedFrom(CLASSNAME::GetStaticTypeObj()) != std::derived_from<T, CLASSNAME>) { \
+                        return false;                                                                                            \
+                    }                                                                                                            \
+                }
+
+            FOR_EACH_K_CLASS_TOKEN_OBJECT_TYPE(CLASS_TOKEN_HANDLER)
+            #undef CLASS_TOKEN_HANDLER
+
+            return true;
+        }
+
+        consteval bool IsEveryAutoObjectInheritanceValid() {
+            #define CLASS_TOKEN_HANDLER(CLASSNAME) if (!IsAutoObjectInheritanceValidImpl<CLASSNAME>()) { return false; }
+            FOR_EACH_K_CLASS_TOKEN_OBJECT_TYPE(CLASS_TOKEN_HANDLER)
+            #undef CLASS_TOKEN_HANDLER
+
+            return true;
+        }
+
+        static_assert(IsEveryAutoObjectInheritanceValid());
+
+        template<typename T>
+        concept IsAutoObjectWithSpecializedGetId = std::derived_from<T, KAutoObjectWithList> && requires (const T &t, const KAutoObjectWithList &l) {
+            { t.GetIdImpl() } -> std::same_as<decltype(l.GetId())>;
+        };
+
+        template<typename T>
+        struct AutoObjectWithListComparatorImpl {
+            using RedBlackKeyType = u64;
+
+            static ALWAYS_INLINE RedBlackKeyType GetRedBlackKey(const RedBlackKeyType &v) { return v; }
+
+            static ALWAYS_INLINE RedBlackKeyType GetRedBlackKey(const KAutoObjectWithList &v) {
+                if constexpr (IsAutoObjectWithSpecializedGetId<T>) {
+                    return static_cast<const T &>(v).GetIdImpl();
+                } else {
+                    return reinterpret_cast<u64>(std::addressof(v));
+                }
+            }
+
+            template<typename U> requires (std::same_as<U, KAutoObjectWithList> || std::same_as<U, RedBlackKeyType>)
+            static ALWAYS_INLINE int Compare(const U &lhs, const KAutoObjectWithList &rhs) {
+                const u64 lid = GetRedBlackKey(lhs);
+                const u64 rid = GetRedBlackKey(rhs);
+
+                if (lid < rid) {
+                    return -1;
+                } else if (lid > rid) {
+                    return 1;
+                } else {
+                    return 0;
+                }
+            }
+        };
+
+        template<typename T>
+        using AutoObjectWithListComparator = AutoObjectWithListComparatorImpl<typename std::conditional<IsAutoObjectWithSpecializedGetId<T>, T, KAutoObjectWithList>::type>;
+
+        template<typename T>
+        using TrueObjectContainerListType = typename KAutoObjectWithListContainer<T>::ListType<AutoObjectWithListComparator<T>>;
+
+        template<typename T>
+        ALWAYS_INLINE TrueObjectContainerListType<T> &GetTrueObjectContainerList(typename KAutoObjectWithListContainer<T>::DummyListType &l) {
+            static_assert(alignof(l) == alignof(impl::TrueObjectContainerListType<T>));
+            static_assert(sizeof(l)  == sizeof(impl::TrueObjectContainerListType<T>));
+            return *reinterpret_cast<TrueObjectContainerListType<T> *>(std::addressof(l));
+        }
+
+    }
+
+    ALWAYS_INLINE void KAutoObject::ScheduleDestruction() {
+        MESOSPHERE_ASSERT_THIS();
+
+        /* Set our object to destroy. */
+        m_next_closed_object = GetCurrentThread().GetClosedObject();
+
+        /* Set ourselves as the thread's next object to destroy. */
+        GetCurrentThread().SetClosedObject(this);
+    }
+
+    template<typename T>
+    class KAutoObjectWithListContainer<T>::ListAccessor : public impl::KAutoObjectWithListContainerBase::ListAccessorImpl<impl::TrueObjectContainerListType<T>> {
+        NON_COPYABLE(ListAccessor);
+        NON_MOVEABLE(ListAccessor);
+        private:
+            using BaseListAccessor = impl::KAutoObjectWithListContainerBase::ListAccessorImpl<impl::TrueObjectContainerListType<T>>;
+        public:
+            explicit ALWAYS_INLINE ListAccessor(KAutoObjectWithListContainer *container) : BaseListAccessor(container, impl::GetTrueObjectContainerList<T>(container->m_dummy_object_list)) { /* ... */ }
+            explicit ALWAYS_INLINE ListAccessor(KAutoObjectWithListContainer &container) : BaseListAccessor(container, impl::GetTrueObjectContainerList<T>(container.m_dummy_object_list)) { /* ... */ }
+
+            ALWAYS_INLINE ~ListAccessor() { /* ... */ }
+    };
+
+
+    template<typename T>
+    ALWAYS_INLINE void KAutoObjectWithListContainer<T>::Register(T *obj) {
+        return this->RegisterImpl(obj, impl::GetTrueObjectContainerList<T>(m_dummy_object_list));
+    }
+
+    template<typename T>
+    ALWAYS_INLINE void KAutoObjectWithListContainer<T>::Unregister(T *obj) {
+        return this->UnregisterImpl(obj, impl::GetTrueObjectContainerList<T>(m_dummy_object_list));
+    }
+
+    template<typename T>
+    ALWAYS_INLINE size_t KAutoObjectWithListContainer<T>::GetOwnedCountChecked(const KProcess *owner) {
+        return this->GetOwnedCountImpl<T>(owner, impl::GetTrueObjectContainerList<T>(m_dummy_object_list));
+    }
+
+    inline u64 KAutoObjectWithList::GetId() const {
+        #define CLASS_TOKEN_HANDLER(CLASSNAME)                                                                      \
+            if constexpr (impl::IsAutoObjectWithSpecializedGetId<CLASSNAME>) {                                      \
+                if (const CLASSNAME * const derived = this->DynamicCast<const CLASSNAME *>(); derived != nullptr) { \
+                    return []<typename T>(const T * const t_derived) ALWAYS_INLINE_LAMBDA -> u64 {                  \
+                        static_assert(std::same_as<T, CLASSNAME>);                                                  \
+                        if constexpr (impl::IsAutoObjectWithSpecializedGetId<CLASSNAME>) {                          \
+                            return impl::AutoObjectWithListComparator<CLASSNAME>::GetRedBlackKey(*t_derived);       \
+                        } else {                                                                                    \
+                            AMS_ASSUME(false);                                                                      \
+                        }                                                                                           \
+                    }(derived);                                                                                     \
+                }                                                                                                   \
+            }
+
+        FOR_EACH_K_CLASS_TOKEN_OBJECT_TYPE(CLASS_TOKEN_HANDLER)
+        #undef CLASS_TOKEN_HANDLER
+
+        return impl::AutoObjectWithListComparator<KAutoObjectWithList>::GetRedBlackKey(*this);
+    }
+
+}

libraries/libmesosphere/include/mesosphere/kern_k_capabilities.hpp

@@ -55,48 +55,16 @@ namespace ams::kern {
                 return static_cast<u32>(type) + 1;
             }
 
-            static constexpr u32 CountTrailingZero(u32 flag) {
-                for (u32 i = 0; i < BITSIZEOF(u32); i++) {
-                    if (flag & (1u << i)) {
-                        return i;
-                    }
-                }
-                return BITSIZEOF(u32);
-            }
-
-            static constexpr u32 GetCapabilityId(CapabilityType type) {
-                const u32 flag = GetCapabilityFlag(type);
-                if (std::is_constant_evaluated()) {
-                    return CountTrailingZero(flag);
-                } else {
-                    return static_cast<u32>(__builtin_ctz(flag));
-                }
-            }
-
             template<size_t Index, size_t Count, typename T = u32>
             using Field = util::BitPack32::Field<Index, Count, T>;
 
             #define DEFINE_FIELD(name, prev, ...) using name = Field<prev::Next, __VA_ARGS__>
 
             template<CapabilityType Type>
-            static constexpr inline u32 CapabilityFlag = []() -> u32 {
-                return static_cast<u32>(Type) + 1;
-            }();
+            static constexpr inline u32 CapabilityFlag = static_cast<u32>(Type) + 1;
 
             template<CapabilityType Type>
-            static constexpr inline u32 CapabilityId = []() -> u32 {
-                const u32 flag = static_cast<u32>(Type) + 1;
-                if (std::is_constant_evaluated()) {
-                    for (u32 i = 0; i < BITSIZEOF(u32); i++) {
-                        if (flag & (1u << i)) {
-                            return i;
-                        }
-                    }
-                    return BITSIZEOF(u32);
-                } else {
-                    return __builtin_ctz(flag);
-                }
-            }();
+            static constexpr inline u32 CapabilityId = util::CountTrailingZeros<u32>(CapabilityFlag<Type>);
 
             struct CorePriority {
                 using IdBits = Field<0, CapabilityId<CapabilityType::CorePriority> + 1>;
@@ -114,7 +82,11 @@ namespace ams::kern {
                 DEFINE_FIELD(Index, Mask,    3);
             };
 
+            #if defined(MESOSPHERE_ENABLE_LARGE_PHYSICAL_ADDRESS_CAPABILITIES)
+            static constexpr u64 PhysicalMapAllowedMask = (1ul << 40) - 1;
+            #else
             static constexpr u64 PhysicalMapAllowedMask = (1ul << 36) - 1;
+            #endif
 
             struct MapRange {
                 using IdBits = Field<0, CapabilityId<CapabilityType::MapRange> + 1>;
@@ -126,9 +98,15 @@ namespace ams::kern {
             struct MapRangeSize {
                 using IdBits = Field<0, CapabilityId<CapabilityType::MapRange> + 1>;
 
-                DEFINE_FIELD(Pages,    IdBits,   20);
+                DEFINE_FIELD(Pages, IdBits, 20);
+
+                #if defined(MESOSPHERE_ENABLE_LARGE_PHYSICAL_ADDRESS_CAPABILITIES)
+                DEFINE_FIELD(AddressHigh, Pages,        4);
+                DEFINE_FIELD(Normal,      AddressHigh,  1, bool);
+                #else
                 DEFINE_FIELD(Reserved, Pages,     4);
                 DEFINE_FIELD(Normal,   Reserved,  1, bool);
+                #endif
             };
 
             struct MapIoPage {
@@ -203,14 +181,14 @@ namespace ams::kern {
                                                        CapabilityFlag<CapabilityType::HandleTable>   |
                                                        CapabilityFlag<CapabilityType::DebugFlags>;
         private:
-            svc::SvcAccessFlagSet m_svc_access_flags{};
-            InterruptFlagSet m_irq_access_flags{};
-            u64 m_core_mask{};
-            u64 m_priority_mask{};
-            util::BitPack32 m_debug_capabilities{0};
-            s32 m_handle_table_size{};
-            util::BitPack32 m_intended_kernel_version{0};
-            u32 m_program_type{};
+            svc::SvcAccessFlagSet m_svc_access_flags;
+            InterruptFlagSet m_irq_access_flags;
+            u64 m_core_mask;
+            u64 m_priority_mask;
+            util::BitPack32 m_debug_capabilities;
+            s32 m_handle_table_size;
+            util::BitPack32 m_intended_kernel_version;
+            u32 m_program_type;
         private:
             constexpr bool SetSvcAllowed(u32 id) {
                 if (AMS_LIKELY(id < m_svc_access_flags.GetCount())) {
@@ -245,7 +223,8 @@ namespace ams::kern {
             Result SetCapabilities(const u32 *caps, s32 num_caps, KProcessPageTable *page_table);
             Result SetCapabilities(svc::KUserPointer<const u32 *> user_caps, s32 num_caps, KProcessPageTable *page_table);
         public:
-            constexpr KCapabilities() = default;
+            constexpr explicit KCapabilities(util::ConstantInitializeTag) : m_svc_access_flags{}, m_irq_access_flags{}, m_core_mask{}, m_priority_mask{}, m_debug_capabilities{0}, m_handle_table_size{}, m_intended_kernel_version{}, m_program_type{} { /* ... */ }
+            KCapabilities() { /* ... */ }
 
             Result Initialize(const u32 *caps, s32 num_caps, KProcessPageTable *page_table);
             Result Initialize(svc::KUserPointer<const u32 *> user_caps, s32 num_caps, KProcessPageTable *page_table);

libraries/libmesosphere/include/mesosphere/kern_k_class_token.hpp

@@ -21,6 +21,35 @@ namespace ams::kern {
 
     class KAutoObject;
 
+    #define FOR_EACH_K_CLASS_TOKEN_OBJECT_TYPE(HANDLER) \
+                HANDLER(KAutoObject)                    \
+                                                        \
+                HANDLER(KSynchronizationObject)         \
+                HANDLER(KReadableEvent)                 \
+                                                        \
+                HANDLER(KInterruptEvent)                \
+                HANDLER(KDebug)                         \
+                HANDLER(KThread)                        \
+                HANDLER(KServerPort)                    \
+                HANDLER(KServerSession)                 \
+                HANDLER(KClientPort)                    \
+                HANDLER(KClientSession)                 \
+                HANDLER(KProcess)                       \
+                HANDLER(KResourceLimit)                 \
+                HANDLER(KLightSession)                  \
+                HANDLER(KPort)                          \
+                HANDLER(KSession)                       \
+                HANDLER(KSharedMemory)                  \
+                HANDLER(KEvent)                         \
+                HANDLER(KLightClientSession)            \
+                HANDLER(KLightServerSession)            \
+                HANDLER(KTransferMemory)                \
+                HANDLER(KDeviceAddressSpace)            \
+                HANDLER(KSessionRequest)                \
+                HANDLER(KCodeMemory)                    \
+                HANDLER(KIoPool)                        \
+                HANDLER(KIoRegion)
+
     class KClassTokenGenerator {
         public:
             using TokenBaseType = u16;
@@ -113,8 +142,11 @@ namespace ams::kern {
                 KIoPool,
                 KIoRegion,
 
+                FinalClassesLast,
+
                 FinalClassesEnd = FinalClassesStart + NumFinalClasses,
             };
+            static_assert(ObjectType::FinalClassesLast <= ObjectType::FinalClassesEnd);
 
             template<typename T>
             static constexpr inline TokenBaseType ClassToken = GetClassToken<T>();
@@ -125,4 +157,37 @@ namespace ams::kern {
     template<typename T>
     static constexpr inline ClassTokenType ClassToken = KClassTokenGenerator::ClassToken<T>;
 
+    namespace impl {
+
+        consteval bool IsKClassTokenGeneratorForEachMacroValid() {
+            auto IsObjectTypeIncludedByMacro = [](KClassTokenGenerator::ObjectType object_type) -> bool {
+                #define CLASS_TOKEN_HANDLER(CLASSNAME) if (object_type == KClassTokenGenerator::ObjectType::CLASSNAME) { return true; }
+                FOR_EACH_K_CLASS_TOKEN_OBJECT_TYPE(CLASS_TOKEN_HANDLER)
+                #undef CLASS_TOKEN_HANDLER
+                return false;
+            };
+
+            if (!IsObjectTypeIncludedByMacro(KClassTokenGenerator::ObjectType::KAutoObject)) {
+                return false;
+            }
+
+            for (auto base = util::ToUnderlying(KClassTokenGenerator::ObjectType::BaseClassesStart); base < util::ToUnderlying(KClassTokenGenerator::ObjectType::BaseClassesEnd); ++base) {
+                if (!IsObjectTypeIncludedByMacro(static_cast<KClassTokenGenerator::ObjectType>(base))) {
+                    return false;
+                }
+            }
+
+            for (auto fin = util::ToUnderlying(KClassTokenGenerator::ObjectType::FinalClassesStart); fin < util::ToUnderlying(KClassTokenGenerator::ObjectType::FinalClassesLast); ++fin) {
+                if (!IsObjectTypeIncludedByMacro(static_cast<KClassTokenGenerator::ObjectType>(fin))) {
+                    return false;
+                }
+            }
+
+            return true;
+        }
+
+        static_assert(IsKClassTokenGeneratorForEachMacroValid());
+
+    }
+
 }

libraries/libmesosphere/include/mesosphere/kern_k_client_port.hpp

@@ -28,12 +28,14 @@ namespace ams::kern {
     class KClientPort final : public KSynchronizationObject {
         MESOSPHERE_AUTOOBJECT_TRAITS(KClientPort, KSynchronizationObject);
         private:
-            std::atomic<s32> m_num_sessions;
-            std::atomic<s32> m_peak_sessions;
+            util::Atomic<s32> m_num_sessions;
+            util::Atomic<s32> m_peak_sessions;
             s32 m_max_sessions;
             KPort *m_parent;
         public:
-            constexpr KClientPort() : m_num_sessions(), m_peak_sessions(), m_max_sessions(), m_parent() { /* ... */ }
+            constexpr explicit KClientPort(util::ConstantInitializeTag) : KSynchronizationObject(util::ConstantInitialize), m_num_sessions(0), m_peak_sessions(0), m_max_sessions(), m_parent() { /* ... */ }
+
+            explicit KClientPort() { /* ... */ }
 
             void Initialize(KPort *parent, s32 max_sessions);
             void OnSessionFinalized();
@@ -41,8 +43,8 @@ namespace ams::kern {
 
             constexpr const KPort *GetParent() const { return m_parent; }
 
-            ALWAYS_INLINE s32 GetNumSessions()  const { return m_num_sessions; }
-            ALWAYS_INLINE s32 GetPeakSessions() const { return m_peak_sessions; }
+            ALWAYS_INLINE s32 GetNumSessions()  const { return m_num_sessions.Load(); }
+            ALWAYS_INLINE s32 GetPeakSessions() const { return m_peak_sessions.Load(); }
             ALWAYS_INLINE s32 GetMaxSessions()  const { return m_max_sessions; }
 
             bool IsLight() const;

libraries/libmesosphere/include/mesosphere/kern_k_client_session.hpp

@@ -27,7 +27,8 @@ namespace ams::kern {
         private:
             KSession *m_parent;
         public:
-            constexpr KClientSession() : m_parent() { /* ... */ }
+            constexpr explicit KClientSession(util::ConstantInitializeTag) : KAutoObject(util::ConstantInitialize), m_parent() { /* ... */ }
+            explicit KClientSession() { /* ... */ }
 
             void Initialize(KSession *parent) {
                 /* Set member variables. */
@@ -35,7 +36,6 @@ namespace ams::kern {
             }
 
             virtual void Destroy() override;
-            static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
             constexpr KSession *GetParent() const { return m_parent; }
 

libraries/libmesosphere/include/mesosphere/kern_k_code_memory.hpp

@@ -36,14 +36,14 @@ namespace ams::kern {
             }
 
             Result Initialize(KProcessAddress address, size_t size);
-            virtual void Finalize() override;
+            void Finalize();
 
             Result Map(KProcessAddress address, size_t size);
             Result Unmap(KProcessAddress address, size_t size);
             Result MapToOwner(KProcessAddress address, size_t size, ams::svc::MemoryPermission perm);
             Result UnmapFromOwner(KProcessAddress address, size_t size);
 
-            virtual bool IsInitialized() const override { return m_is_initialized; }
+            bool IsInitialized() const { return m_is_initialized; }
             static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
             KProcess *GetOwner() const { return m_owner; }

libraries/libmesosphere/include/mesosphere/kern_k_condition_variable.hpp

@@ -26,7 +26,7 @@ namespace ams::kern {
         private:
             ThreadTree m_tree;
         public:
-            constexpr KConditionVariable() : m_tree() { /* ... */ }
+            constexpr KConditionVariable() = default;
 
             /* Arbitration. */
             static Result SignalToAddress(KProcessAddress addr);

libraries/libmesosphere/include/mesosphere/kern_k_debug_base.hpp

@@ -25,35 +25,20 @@ namespace ams::kern {
     class KDebugBase : public KSynchronizationObject {
         protected:
             using DebugEventList = util::IntrusiveListBaseTraits<KEventInfo>::ListType;
-        private:
-            class ProcessHolder {
-                private:
-                    friend class KDebugBase;
-                private:
-                    KProcess *m_process;
-                    std::atomic<u32> m_ref_count;
-                private:
-                    explicit ProcessHolder() : m_process(nullptr) { /* ... */ }
-
-                    void Attach(KProcess *process);
-                    void Detach();
-
-                    bool Open();
-                    void Close();
-            };
         private:
             DebugEventList m_event_info_list;
             u32 m_continue_flags;
-            ProcessHolder m_process_holder;
+            KSharedAutoObject<KProcess> m_process_holder;
             KLightLock m_lock;
             KProcess::State m_old_process_state;
             bool m_is_attached;
         public:
-            explicit KDebugBase() : m_event_info_list(), m_process_holder(), m_lock() { /* ... */ }
+            explicit KDebugBase() { /* ... */ }
         protected:
             bool Is64Bit() const;
         public:
             void Initialize();
+            void Finalize();
 
             Result Attach(KProcess *process);
             Result BreakProcess();
@@ -68,9 +53,6 @@ namespace ams::kern {
             Result GetThreadContext(ams::svc::ThreadContext *out, u64 thread_id, u32 context_flags);
             Result SetThreadContext(const ams::svc::ThreadContext &ctx, u64 thread_id, u32 context_flags);
 
-            virtual Result GetThreadContextImpl(ams::svc::ThreadContext *out, KThread *thread, u32 context_flags) = 0;
-            virtual Result SetThreadContextImpl(const ams::svc::ThreadContext &ctx, KThread *thread, u32 context_flags) = 0;
-
             Result GetRunningThreadInfo(ams::svc::LastThreadContext *out_context, u64 *out_thread_id);
 
             Result GetDebugEventInfo(ams::svc::lp64::DebugEventInfo *out);
@@ -89,7 +71,7 @@ namespace ams::kern {
             }
 
             ALWAYS_INLINE KProcess *GetProcessUnsafe() const {
-                return m_process_holder.m_process;
+                return m_process_holder.Get();
             }
         private:
             void PushDebugEvent(ams::svc::DebugEvent event, uintptr_t param0 = 0, uintptr_t param1 = 0, uintptr_t param2 = 0, uintptr_t param3 = 0, uintptr_t param4 = 0);
@@ -98,8 +80,10 @@ namespace ams::kern {
             template<typename T> requires (std::same_as<T, ams::svc::lp64::DebugEventInfo> || std::same_as<T, ams::svc::ilp32::DebugEventInfo>)
             Result GetDebugEventInfoImpl(T *out);
         public:
-            virtual void OnFinalizeSynchronizationObject() override;
             virtual bool IsSignaled() const override;
+        private:
+            /* NOTE: This is public/virtual override in Nintendo's kernel. */
+            void OnFinalizeSynchronizationObject();
         private:
             static Result ProcessDebugEvent(ams::svc::DebugEvent event, uintptr_t param0, uintptr_t param1, uintptr_t param2, uintptr_t param3, uintptr_t param4);
         public:

libraries/libmesosphere/include/mesosphere/kern_k_device_address_space.hpp

@@ -30,12 +30,12 @@ namespace ams::kern {
             u64 m_space_size;
             bool m_is_initialized;
         public:
-            constexpr KDeviceAddressSpace() : m_lock(), m_table(), m_space_address(), m_space_size(), m_is_initialized() { /* ... */ }
+            explicit KDeviceAddressSpace() : m_is_initialized(false) { /* ... */ }
 
             Result Initialize(u64 address, u64 size);
-            virtual void Finalize() override;
+            void Finalize();
 
-            virtual bool IsInitialized() const override { return m_is_initialized; }
+            bool IsInitialized() const { return m_is_initialized; }
             static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
             Result Attach(ams::svc::DeviceName device_name);

libraries/libmesosphere/include/mesosphere/kern_k_dynamic_page_manager.hpp

@@ -40,7 +40,7 @@ namespace ams::kern {
             KVirtualAddress m_address;
             size_t m_size;
         public:
-            KDynamicPageManager() : m_lock(), m_page_bitmap(), m_used(), m_peak(), m_count(), m_address(), m_size() { /* ... */ }
+            KDynamicPageManager() : m_lock(), m_page_bitmap(), m_used(), m_peak(), m_count(), m_address(Null<KVirtualAddress>), m_size() { /* ... */ }
 
             Result Initialize(KVirtualAddress memory, size_t sz) {
                 /* We need to have positive size. */

libraries/libmesosphere/include/mesosphere/kern_k_dynamic_resource_manager.hpp

@@ -46,7 +46,7 @@ namespace ams::kern {
                 return m_slab_heap->Allocate(m_page_allocator);
             }
 
-            void Free(T *t) const {
+            ALWAYS_INLINE void Free(T *t) const {
                 m_slab_heap->Free(t);
             }
     };

libraries/libmesosphere/include/mesosphere/kern_k_dynamic_slab_heap.hpp

@@ -29,19 +29,19 @@ namespace ams::kern {
         private:
             using PageBuffer = KDynamicPageManager::PageBuffer;
         private:
-            std::atomic<size_t> m_used{};
-            std::atomic<size_t> m_peak{};
-            std::atomic<size_t> m_count{};
-            KVirtualAddress m_address{};
+            util::Atomic<size_t> m_used{0};
+            util::Atomic<size_t> m_peak{0};
+            util::Atomic<size_t> m_count{0};
+            KVirtualAddress m_address{Null<KVirtualAddress>};
             size_t m_size{};
         public:
             constexpr KDynamicSlabHeap() = default;
 
             constexpr ALWAYS_INLINE KVirtualAddress GetAddress() const { return m_address; }
             constexpr ALWAYS_INLINE size_t GetSize() const { return m_size; }
-            constexpr ALWAYS_INLINE size_t GetUsed() const { return m_used.load(); }
-            constexpr ALWAYS_INLINE size_t GetPeak() const { return m_peak.load(); }
-            constexpr ALWAYS_INLINE size_t GetCount() const { return m_count.load(); }
+            constexpr ALWAYS_INLINE size_t GetUsed() const { return m_used.Load(); }
+            constexpr ALWAYS_INLINE size_t GetPeak() const { return m_peak.Load(); }
+            constexpr ALWAYS_INLINE size_t GetCount() const { return m_count.Load(); }
 
             constexpr ALWAYS_INLINE bool IsInRange(KVirtualAddress addr) const {
                 return this->GetAddress() <= addr && addr <= this->GetAddress() + this->GetSize() - 1;
@@ -58,7 +58,7 @@ namespace ams::kern {
                 KSlabHeapImpl::Initialize();
 
                 /* Allocate until we have the correct number of objects. */
-                while (m_count.load() < num_objects) {
+                while (m_count.Load() < num_objects) {
                     auto *allocated = reinterpret_cast<T *>(page_allocator->Allocate());
                     MESOSPHERE_ABORT_UNLESS(allocated != nullptr);
 
@@ -66,7 +66,7 @@ namespace ams::kern {
                         KSlabHeapImpl::Free(allocated + i);
                     }
 
-                    m_count.fetch_add(sizeof(PageBuffer) / sizeof(T));
+                    m_count += sizeof(PageBuffer) / sizeof(T);
                 }
             }
 
@@ -89,7 +89,7 @@ namespace ams::kern {
                             for (size_t i = 1; i < sizeof(PageBuffer) / sizeof(T); i++) {
                                 KSlabHeapImpl::Free(allocated + i);
                             }
-                            m_count.fetch_add(sizeof(PageBuffer) / sizeof(T));
+                            m_count += sizeof(PageBuffer) / sizeof(T);
                         }
                     }
                 }
@@ -99,10 +99,10 @@ namespace ams::kern {
                     std::construct_at(allocated);
 
                     /* Update our tracking. */
-                    size_t used = m_used.fetch_add(1) + 1;
-                    size_t peak = m_peak.load();
+                    const size_t used = ++m_used;
+                    size_t peak = m_peak.Load();
                     while (peak < used) {
-                        if (m_peak.compare_exchange_weak(peak, used, std::memory_order_relaxed)) {
+                        if (m_peak.CompareExchangeWeak<std::memory_order_relaxed>(peak, used)) {
                             break;
                         }
                     }
@@ -113,7 +113,7 @@ namespace ams::kern {
 
             ALWAYS_INLINE void Free(T *t) {
                 KSlabHeapImpl::Free(t);
-                m_used.fetch_sub(1);
+                --m_used;
             }
     };
 

libraries/libmesosphere/include/mesosphere/kern_k_event.hpp

@@ -29,21 +29,24 @@ namespace ams::kern {
             bool m_initialized;
             bool m_readable_event_destroyed;
         public:
-            constexpr KEvent()
-                : m_readable_event(), m_owner(), m_initialized(), m_readable_event_destroyed()
+            constexpr explicit KEvent(util::ConstantInitializeTag)
+                : KAutoObjectWithSlabHeapAndContainer<KEvent, KAutoObjectWithList, true>(util::ConstantInitialize),
+                  m_readable_event(util::ConstantInitialize), m_owner(), m_initialized(), m_readable_event_destroyed()
             {
                 /* ... */
             }
 
-            void Initialize();
-            virtual void Finalize() override;
+            explicit KEvent() : m_readable_event(), m_owner(), m_initialized(), m_readable_event_destroyed() { /* ... */ }
 
-            virtual bool IsInitialized() const override { return m_initialized; }
-            virtual uintptr_t GetPostDestroyArgument() const override { return reinterpret_cast<uintptr_t>(m_owner); }
+            void Initialize();
+            void Finalize();
+
+            bool IsInitialized() const { return m_initialized; }
+            uintptr_t GetPostDestroyArgument() const { return reinterpret_cast<uintptr_t>(m_owner); }
 
             static void PostDestroy(uintptr_t arg);
 
-            virtual KProcess *GetOwner() const override { return m_owner; }
+            KProcess *GetOwner() const { return m_owner; }
 
             KReadableEvent &GetReadableEvent() { return m_readable_event; }
 

libraries/libmesosphere/include/mesosphere/kern_k_handle_table.hpp

@@ -63,16 +63,16 @@ namespace ams::kern {
         private:
             EntryInfo m_entry_infos[MaxTableSize];
             KAutoObject *m_objects[MaxTableSize];
+            mutable KSpinLock m_lock;
             s32 m_free_head_index;
             u16 m_table_size;
             u16 m_max_count;
             u16 m_next_linear_id;
             u16 m_count;
-            mutable KSpinLock m_lock;
         public:
-            constexpr KHandleTable() :
-                m_entry_infos(), m_objects(), m_free_head_index(-1), m_table_size(0), m_max_count(0), m_next_linear_id(MinLinearId), m_count(0), m_lock()
-            { MESOSPHERE_ASSERT_THIS(); }
+            constexpr explicit KHandleTable(util::ConstantInitializeTag) : m_entry_infos(), m_objects(), m_lock(), m_free_head_index(-1), m_table_size(), m_max_count(), m_next_linear_id(MinLinearId), m_count() { /* ... */ }
+
+            explicit KHandleTable() : m_lock(), m_free_head_index(-1), m_count() { MESOSPHERE_ASSERT_THIS(); }
 
             constexpr NOINLINE Result Initialize(s32 size) {
                 MESOSPHERE_ASSERT_THIS();

libraries/libmesosphere/include/mesosphere/kern_k_interrupt_event.hpp

@@ -31,14 +31,25 @@ namespace ams::kern {
             s32 m_core_id;
             bool m_is_initialized;
         public:
-            constexpr KInterruptEvent() : m_interrupt_id(-1), m_core_id(-1), m_is_initialized(false) { /* ... */ }
+            constexpr explicit KInterruptEvent(util::ConstantInitializeTag) : KAutoObjectWithSlabHeapAndContainer<KInterruptEvent, KReadableEvent>(util::ConstantInitialize), m_interrupt_id(-1), m_core_id(-1), m_is_initialized(false) { /* ... */ }
+
+            explicit KInterruptEvent() : m_interrupt_id(-1), m_is_initialized(false) { /* ... */ }
 
             Result Initialize(int32_t interrupt_name, ams::svc::InterruptType type);
-            virtual void Finalize() override;
+            void Finalize();
 
-            virtual Result Reset() override;
+            Result Reset();
 
-            virtual bool IsInitialized() const override { return m_is_initialized; }
+            Result Clear() {
+                MESOSPHERE_ASSERT_THIS();
+
+                /* Try to perform a reset, succeeding unconditionally. */
+                this->Reset();
+
+                return ResultSuccess();
+            }
+
+            bool IsInitialized() const { return m_is_initialized; }
 
             static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 

libraries/libmesosphere/include/mesosphere/kern_k_io_pool.hpp

@@ -33,14 +33,14 @@ namespace ams::kern {
         public:
             static bool IsValidIoPoolType(ams::svc::IoPoolType pool_type);
         public:
-            explicit KIoPool() : m_lock(), m_io_region_list(), m_is_initialized(false) {
+            explicit KIoPool() : m_is_initialized(false) {
                 /* ... */
             }
 
             Result Initialize(ams::svc::IoPoolType pool_type);
-            virtual void Finalize() override;
+            void Finalize();
 
-            virtual bool IsInitialized() const override { return m_is_initialized; }
+            bool IsInitialized() const { return m_is_initialized; }
             static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
             Result AddIoRegion(KIoRegion *region);

libraries/libmesosphere/include/mesosphere/kern_k_io_region.hpp

@@ -40,16 +40,12 @@ namespace ams::kern {
             util::IntrusiveListNode m_process_list_node;
             util::IntrusiveListNode m_pool_list_node;
         public:
-            explicit KIoRegion()
-                : m_lock(), m_pool(nullptr), m_is_initialized(false), m_process_list_node(), m_pool_list_node()
-            {
-                /* ... */
-            }
+            explicit KIoRegion() : m_pool(nullptr), m_is_initialized(false) { /* ... */ }
 
             Result Initialize(KIoPool *pool, KPhysicalAddress phys_addr, size_t size, ams::svc::MemoryMapping mapping, ams::svc::MemoryPermission perm);
-            virtual void Finalize() override;
+            void Finalize();
 
-            virtual bool IsInitialized() const override { return m_is_initialized; }
+            bool IsInitialized() const { return m_is_initialized; }
             static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
             Result Map(KProcessAddress address, size_t size, ams::svc::MemoryPermission map_perm);

libraries/libmesosphere/include/mesosphere/kern_k_light_client_session.hpp

@@ -26,7 +26,7 @@ namespace ams::kern {
         private:
             KLightSession *m_parent;
         public:
-            constexpr KLightClientSession() : m_parent() { /* ... */ }
+            explicit KLightClientSession() { /* ... */ }
 
             void Initialize(KLightSession *parent) {
                 /* Set member variables. */
@@ -34,7 +34,6 @@ namespace ams::kern {
             }
 
             virtual void Destroy() override;
-            static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
             constexpr const KLightSession *GetParent() const { return m_parent; }
 

libraries/libmesosphere/include/mesosphere/kern_k_light_condition_variable.hpp

@@ -26,7 +26,9 @@ namespace ams::kern {
         private:
             KThread::WaiterList m_wait_list;
         public:
-            constexpr ALWAYS_INLINE KLightConditionVariable() : m_wait_list() { /* ... */ }
+            constexpr explicit ALWAYS_INLINE KLightConditionVariable(util::ConstantInitializeTag) : m_wait_list() { /* ... */ }
+
+            explicit ALWAYS_INLINE KLightConditionVariable() { /* ... */ }
         public:
             void Wait(KLightLock *lock, s64 timeout = -1ll, bool allow_terminating_thread = true);
             void Broadcast();

libraries/libmesosphere/include/mesosphere/kern_k_light_lock.hpp

@@ -23,30 +23,25 @@ namespace ams::kern {
 
     class KLightLock {
         private:
-            std::atomic<uintptr_t> m_tag;
+            util::Atomic<uintptr_t> m_tag;
         public:
-            constexpr KLightLock() : m_tag(0) { /* ... */ }
+            constexpr ALWAYS_INLINE KLightLock() : m_tag(0) { /* ... */ }
 
             void Lock() {
                 MESOSPHERE_ASSERT_THIS();
 
                 const uintptr_t cur_thread = reinterpret_cast<uintptr_t>(GetCurrentThreadPointer());
-                const uintptr_t cur_thread_tag = (cur_thread | 1);
 
                 while (true) {
-                    uintptr_t old_tag = m_tag.load(std::memory_order_relaxed);
+                    uintptr_t old_tag = m_tag.Load<std::memory_order_relaxed>();
 
-                    while (!m_tag.compare_exchange_weak(old_tag, (old_tag == 0) ? cur_thread : old_tag | 1, std::memory_order_acquire)) {
-                        if ((old_tag | 1) == cur_thread_tag) {
-                            return;
-                        }
+                    while (!m_tag.CompareExchangeWeak<std::memory_order_acquire>(old_tag, (old_tag == 0) ? cur_thread : (old_tag | 1))) {
+                        /* ... */
                     }
 
-                    if ((old_tag == 0) || ((old_tag | 1) == cur_thread_tag)) {
+                    if (old_tag == 0 || this->LockSlowPath(old_tag | 1, cur_thread)) {
                         break;
                     }
-
-                    this->LockSlowPath(old_tag | 1, cur_thread);
                 }
             }
 
@@ -54,19 +49,18 @@ namespace ams::kern {
                 MESOSPHERE_ASSERT_THIS();
 
                 const uintptr_t cur_thread = reinterpret_cast<uintptr_t>(GetCurrentThreadPointer());
+
                 uintptr_t expected = cur_thread;
-                do {
-                    if (expected != cur_thread) {
-                        return this->UnlockSlowPath(cur_thread);
-                    }
-                } while (!m_tag.compare_exchange_weak(expected, 0, std::memory_order_release));
+                if (!m_tag.CompareExchangeStrong<std::memory_order_release>(expected, 0)) {
+                    this->UnlockSlowPath(cur_thread);
+                }
             }
 
-            void LockSlowPath(uintptr_t owner, uintptr_t cur_thread);
+            bool LockSlowPath(uintptr_t owner, uintptr_t cur_thread);
             void UnlockSlowPath(uintptr_t cur_thread);
 
-            ALWAYS_INLINE bool IsLocked() const { return m_tag.load() != 0; }
-            ALWAYS_INLINE bool IsLockedByCurrentThread() const { return (m_tag.load() | 0x1ul) == (reinterpret_cast<uintptr_t>(GetCurrentThreadPointer()) | 0x1ul); }
+            ALWAYS_INLINE bool IsLocked() const { return m_tag.Load() != 0; }
+            ALWAYS_INLINE bool IsLockedByCurrentThread() const { return (m_tag.Load() | 0x1ul) == (reinterpret_cast<uintptr_t>(GetCurrentThreadPointer()) | 0x1ul); }
     };
 
     using KScopedLightLock = KScopedLock<KLightLock>;

libraries/libmesosphere/include/mesosphere/kern_k_light_server_session.hpp

@@ -32,7 +32,7 @@ namespace ams::kern {
             u64 m_server_thread_id;
             KThread *m_server_thread;
         public:
-            constexpr KLightServerSession() : m_parent(), m_request_list(), m_current_request(), m_server_thread_id(), m_server_thread() { /* ... */ }
+            explicit KLightServerSession() : m_current_request(nullptr), m_server_thread_id(std::numeric_limits<u64>::max()), m_server_thread() { /* ... */ }
 
             void Initialize(KLightSession *parent) {
                 /* Set member variables. */
@@ -40,7 +40,6 @@ namespace ams::kern {
             }
 
             virtual void Destroy() override;
-            static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
             constexpr const KLightSession *GetParent() const { return m_parent; }
 

libraries/libmesosphere/include/mesosphere/kern_k_light_session.hpp

@@ -46,17 +46,13 @@ namespace ams::kern {
             KProcess *m_process;
             bool m_initialized;
         public:
-            constexpr KLightSession()
-                : m_server(), m_client(), m_state(State::Invalid), m_port(), m_name(), m_process(), m_initialized()
-            {
-                /* ... */
-            }
+            explicit KLightSession() : m_state(State::Invalid), m_process(), m_initialized() { /* ... */ }
 
             void Initialize(KClientPort *client_port, uintptr_t name);
-            virtual void Finalize() override;
+            void Finalize();
 
-            virtual bool IsInitialized() const override { return m_initialized; }
-            virtual uintptr_t GetPostDestroyArgument() const override { return reinterpret_cast<uintptr_t>(m_process); }
+            bool IsInitialized() const { return m_initialized; }
+            uintptr_t GetPostDestroyArgument() const { return reinterpret_cast<uintptr_t>(m_process); }
 
             static void PostDestroy(uintptr_t arg);
 

libraries/libmesosphere/include/mesosphere/kern_k_memory_block.hpp

@@ -349,28 +349,30 @@ namespace ams::kern {
                 };
             }
         public:
-            constexpr KMemoryBlock()
-                : m_device_disable_merge_left_count(), m_device_disable_merge_right_count(), m_address(), m_num_pages(), m_memory_state(KMemoryState_None), m_ipc_lock_count(), m_device_use_count(), m_ipc_disable_merge_count(), m_permission(), m_original_permission(), m_attribute(), m_disable_merge_attribute()
-            {
-                /* ... */
-            }
+            explicit KMemoryBlock() { /* ... */ }
 
-            constexpr KMemoryBlock(KProcessAddress addr, size_t np, KMemoryState ms, KMemoryPermission p, KMemoryAttribute attr)
-                : m_device_disable_merge_left_count(), m_device_disable_merge_right_count(), m_address(addr), m_num_pages(np), m_memory_state(ms), m_ipc_lock_count(0), m_device_use_count(0), m_ipc_disable_merge_count(), m_permission(p), m_original_permission(KMemoryPermission_None), m_attribute(attr), m_disable_merge_attribute()
+            constexpr KMemoryBlock(util::ConstantInitializeTag, KProcessAddress addr, size_t np, KMemoryState ms, KMemoryPermission p, KMemoryAttribute attr)
+                : util::IntrusiveRedBlackTreeBaseNode<KMemoryBlock>(util::ConstantInitialize), m_device_disable_merge_left_count(),
+                  m_device_disable_merge_right_count(), m_address(addr), m_num_pages(np), m_memory_state(ms), m_ipc_lock_count(0),
+                  m_device_use_count(0), m_ipc_disable_merge_count(), m_permission(p), m_original_permission(KMemoryPermission_None),
+                  m_attribute(attr), m_disable_merge_attribute()
             {
                 /* ... */
             }
 
             constexpr void Initialize(KProcessAddress addr, size_t np, KMemoryState ms, KMemoryPermission p, KMemoryAttribute attr) {
                 MESOSPHERE_ASSERT_THIS();
-                m_address             = addr;
-                m_num_pages           = np;
-                m_memory_state        = ms;
-                m_ipc_lock_count      = 0;
-                m_device_use_count    = 0;
-                m_permission          = p;
-                m_original_permission = KMemoryPermission_None;
-                m_attribute           = attr;
+                m_device_disable_merge_left_count  = 0;
+                m_device_disable_merge_right_count = 0;
+                m_address                          = addr;
+                m_num_pages                        = np;
+                m_memory_state                     = ms;
+                m_ipc_lock_count                   = 0;
+                m_device_use_count                 = 0;
+                m_permission                       = p;
+                m_original_permission              = KMemoryPermission_None;
+                m_attribute                        = attr;
+                m_disable_merge_attribute          = KMemoryBlockDisableMergeAttribute_None;
             }
 
             constexpr bool HasProperties(KMemoryState s, KMemoryPermission p, KMemoryAttribute a) const {

libraries/libmesosphere/include/mesosphere/kern_k_memory_block_manager.hpp

@@ -88,7 +88,9 @@ namespace ams::kern {
         private:
             void CoalesceForUpdate(KMemoryBlockManagerUpdateAllocator *allocator, KProcessAddress address, size_t num_pages);
         public:
-            constexpr KMemoryBlockManager() : m_memory_block_tree(), m_start_address(), m_end_address() { /* ... */ }
+            constexpr explicit KMemoryBlockManager(util::ConstantInitializeTag) : m_memory_block_tree(), m_start_address(Null<KProcessAddress>), m_end_address(Null<KProcessAddress>) { /* ... */ }
+
+            explicit KMemoryBlockManager() { /* ... */ }
 
             iterator end() { return m_memory_block_tree.end(); }
             const_iterator end() const { return m_memory_block_tree.end(); }
@@ -105,7 +107,7 @@ namespace ams::kern {
             void UpdateIfMatch(KMemoryBlockManagerUpdateAllocator *allocator, KProcessAddress address, size_t num_pages, KMemoryState test_state, KMemoryPermission test_perm, KMemoryAttribute test_attr, KMemoryState state, KMemoryPermission perm, KMemoryAttribute attr);
 
             iterator FindIterator(KProcessAddress address) const {
-                return m_memory_block_tree.find(KMemoryBlock(address, 1, KMemoryState_Free, KMemoryPermission_None, KMemoryAttribute_None));
+                return m_memory_block_tree.find(KMemoryBlock(util::ConstantInitialize, address, 1, KMemoryState_Free, KMemoryPermission_None, KMemoryAttribute_None));
             }
 
             const KMemoryBlock *FindBlock(KProcessAddress address) const {

libraries/libmesosphere/include/mesosphere/kern_k_memory_layout.hpp

@@ -20,6 +20,8 @@
 
 #if defined(ATMOSPHERE_BOARD_NINTENDO_NX)
     #include <mesosphere/board/nintendo/nx/kern_k_memory_layout.hpp>
+#elif defined(ATMOSPHERE_BOARD_QEMU_VIRT)
+    #include <mesosphere/board/qemu/virt/kern_k_memory_layout.hpp>
 #else
     #error "Unknown board for KMemoryLayout"
 #endif
@@ -54,12 +56,12 @@ namespace ams::kern {
 
     class KMemoryLayout {
         private:
-            static /* constinit */ inline uintptr_t s_linear_phys_to_virt_diff;
-            static /* constinit */ inline uintptr_t s_linear_virt_to_phys_diff;
-            static /* constinit */ inline KMemoryRegionTree s_virtual_tree;
-            static /* constinit */ inline KMemoryRegionTree s_physical_tree;
-            static /* constinit */ inline KMemoryRegionTree s_virtual_linear_tree;
-            static /* constinit */ inline KMemoryRegionTree s_physical_linear_tree;
+            static constinit inline uintptr_t s_linear_phys_to_virt_diff;
+            static constinit inline uintptr_t s_linear_virt_to_phys_diff;
+            static constinit inline KMemoryRegionTree s_virtual_tree;
+            static constinit inline KMemoryRegionTree s_physical_tree;
+            static constinit inline KMemoryRegionTree s_virtual_linear_tree;
+            static constinit inline KMemoryRegionTree s_physical_linear_tree;
         private:
             template<typename AddressType> requires IsKTypedAddress<AddressType>
             static ALWAYS_INLINE bool IsTypedAddress(const KMemoryRegion *&region, AddressType address, KMemoryRegionTree &tree, KMemoryRegionType type) {
@@ -210,13 +212,17 @@ namespace ams::kern {
             static NOINLINE auto GetKernelPageTableHeapRegionPhysicalExtents() { return GetPhysicalMemoryRegionTree().GetDerivedRegionExtents(KMemoryRegionType_DramKernelPtHeap); }
             static NOINLINE auto GetKernelInitPageTableRegionPhysicalExtents() { return GetPhysicalMemoryRegionTree().GetDerivedRegionExtents(KMemoryRegionType_DramKernelInitPt); }
 
-            static NOINLINE auto GetKernelPoolManagementRegionPhysicalExtents() { return GetPhysicalMemoryRegionTree().GetDerivedRegionExtents(KMemoryRegionType_DramPoolManagement); }
             static NOINLINE auto GetKernelPoolPartitionRegionPhysicalExtents() { return GetPhysicalMemoryRegionTree().GetDerivedRegionExtents(KMemoryRegionType_DramPoolPartition); }
+            static NOINLINE auto GetKernelPoolManagementRegionPhysicalExtents() { return GetPhysicalMemoryRegionTree().GetDerivedRegionExtents(KMemoryRegionType_DramPoolManagement); }
             static NOINLINE auto GetKernelSystemPoolRegionPhysicalExtents() { return GetPhysicalMemoryRegionTree().GetDerivedRegionExtents(KMemoryRegionType_DramSystemPool); }
             static NOINLINE auto GetKernelSystemNonSecurePoolRegionPhysicalExtents() { return GetPhysicalMemoryRegionTree().GetDerivedRegionExtents(KMemoryRegionType_DramSystemNonSecurePool); }
             static NOINLINE auto GetKernelAppletPoolRegionPhysicalExtents() { return GetPhysicalMemoryRegionTree().GetDerivedRegionExtents(KMemoryRegionType_DramAppletPool); }
             static NOINLINE auto GetKernelApplicationPoolRegionPhysicalExtents() { return GetPhysicalMemoryRegionTree().GetDerivedRegionExtents(KMemoryRegionType_DramApplicationPool); }
 
+            static NOINLINE bool HasKernelSystemNonSecurePoolRegion() { return GetPhysicalMemoryRegionTree().FindFirstDerived(KMemoryRegionType_DramSystemNonSecurePool) != nullptr; }
+            static NOINLINE bool HasKernelAppletPoolRegion() { return GetPhysicalMemoryRegionTree().FindFirstDerived(KMemoryRegionType_DramAppletPool) != nullptr; }
+            static NOINLINE bool HasKernelApplicationPoolRegion() { return GetPhysicalMemoryRegionTree().FindFirstDerived(KMemoryRegionType_DramApplicationPool) != nullptr; }
+
             static NOINLINE auto GetKernelTraceBufferRegionPhysicalExtents() { return GetPhysicalMemoryRegionTree().GetDerivedRegionExtents(KMemoryRegionType_KernelTraceBuffer); }
     };
 

libraries/libmesosphere/include/mesosphere/kern_k_memory_manager.hpp

@@ -68,7 +68,7 @@ namespace ams::kern {
                     Impl *m_next;
                     Impl *m_prev;
                 public:
-                    Impl() : m_heap(), m_page_reference_counts(), m_management_region(), m_pool(), m_next(), m_prev() { /* ... */ }
+                    Impl() : m_heap(), m_page_reference_counts(), m_management_region(Null<KVirtualAddress>), m_pool(), m_next(), m_prev() { /* ... */ }
 
                     size_t Initialize(KPhysicalAddress address, size_t size, KVirtualAddress management, KVirtualAddress management_end, Pool p);
 

libraries/libmesosphere/include/mesosphere/kern_k_memory_region.hpp

@@ -43,13 +43,14 @@ namespace ams::kern {
                 }
             }
         public:
-            constexpr ALWAYS_INLINE KMemoryRegion() : m_address(0), m_pair_address(0), m_last_address(0), m_attributes(0), m_type_id(0) { /* ... */ }
-            constexpr ALWAYS_INLINE KMemoryRegion(uintptr_t a, size_t la, uintptr_t p, u32 r, u32 t) :
+            constexpr ALWAYS_INLINE KMemoryRegion() : util::IntrusiveRedBlackTreeBaseNode<KMemoryRegion>(util::ConstantInitialize), m_address(0), m_pair_address(0), m_last_address(0), m_attributes(0), m_type_id(0) { /* ... */ }
+
+            ALWAYS_INLINE KMemoryRegion(uintptr_t a, size_t la, uintptr_t p, u32 r, u32 t) :
                 m_address(a), m_pair_address(p), m_last_address(la), m_attributes(r), m_type_id(t)
             {
                 /* ... */
             }
-            constexpr ALWAYS_INLINE KMemoryRegion(uintptr_t a, size_t la, u32 r, u32 t) : KMemoryRegion(a, la, std::numeric_limits<uintptr_t>::max(), r, t) { /* ... */ }
+            ALWAYS_INLINE KMemoryRegion(uintptr_t a, size_t la, u32 r, u32 t) : KMemoryRegion(a, la, std::numeric_limits<uintptr_t>::max(), r, t) { /* ... */ }
         private:
             constexpr ALWAYS_INLINE void Reset(uintptr_t a, uintptr_t la, uintptr_t p, u32 r, u32 t) {
                 m_address      = a;

libraries/libmesosphere/include/mesosphere/kern_k_object_name.hpp

@@ -29,8 +29,6 @@ namespace ams::kern {
         private:
             char m_name[NameLengthMax];
             KAutoObject *m_object;
-        public:
-            constexpr KObjectName() : m_name(), m_object() { /* ... */ }
         public:
             static Result NewFromName(KAutoObject *obj, const char *name);
             static Result Delete(KAutoObject *obj, const char *name);

libraries/libmesosphere/include/mesosphere/kern_k_page_bitmap.hpp

@@ -43,44 +43,25 @@ namespace ams::kern {
                         return rnd_bit;
                     }
                 public:
-                    RandomBitGenerator() : m_rng(), m_entropy(), m_bits_available() {
+                    RandomBitGenerator() : m_entropy(), m_bits_available() {
                         m_rng.Initialize(static_cast<u32>(KSystemControl::GenerateRandomU64()));
                     }
 
                     size_t SelectRandomBit(u64 bitmap) {
-                        u64 selected     = 0;
+                        u64 selected = 0;
 
-                        u64 cur_num_bits = BITSIZEOF(bitmap) / 2;
-                        u64 cur_mask     = (1ull << cur_num_bits) - 1;
+                        for (size_t cur_num_bits = BITSIZEOF(bitmap) / 2; cur_num_bits != 0; cur_num_bits /= 2) {
+                            const u64 high = (bitmap >> cur_num_bits);
+                            const u64 low  = (bitmap & (~(UINT64_C(0xFFFFFFFFFFFFFFFF) << cur_num_bits)));
 
-                        while (cur_num_bits) {
-                            const u64 low  = (bitmap >> 0) & cur_mask;
-                            const u64 high = (bitmap >> cur_num_bits) & cur_mask;
-
-                            bool choose_low;
-                            if (high == 0) {
-                                /* If only low val is set, choose low. */
-                                choose_low = true;
-                            } else if (low == 0) {
-                                /* If only high val is set, choose high. */
-                                choose_low = false;
-                            } else {
-                                /* If both are set, choose random. */
-                                choose_low = this->GenerateRandomBit();
-                            }
-
-                            /* If we chose low, proceed with low. */
-                            if (choose_low) {
-                                bitmap    = low;
-                                selected += 0;
-                            } else {
+                            /* Choose high if we have high and (don't have low or select high randomly). */
+                            if (high && (low == 0 || this->GenerateRandomBit())) {
                                 bitmap    = high;
                                 selected += cur_num_bits;
+                            } else {
+                                bitmap    = low;
+                                selected += 0;
                             }
-
-                            /* Proceed. */
-                            cur_num_bits /= 2;
-                            cur_mask >>= cur_num_bits;
                         }
 
                         return selected;

libraries/libmesosphere/include/mesosphere/kern_k_page_group.hpp

@@ -28,11 +28,11 @@ namespace ams::kern {
         private:
             friend class KPageGroup;
         private:
-            KBlockInfo *m_next{};
-            u32 m_page_index{};
-            u32 m_num_pages{};
+            KBlockInfo *m_next;
+            u32 m_page_index;
+            u32 m_num_pages;
         public:
-            constexpr KBlockInfo() = default;
+            KBlockInfo() : m_next(nullptr) { /* ... */ }
 
             constexpr ALWAYS_INLINE void Initialize(KPhysicalAddress addr, size_t np) {
                 MESOSPHERE_ASSERT(util::IsAligned(GetInteger(addr), PageSize));

libraries/libmesosphere/include/mesosphere/kern_k_page_heap.hpp

@@ -59,7 +59,7 @@ namespace ams::kern {
                     size_t m_block_shift;
                     size_t m_next_block_shift;
                 public:
-                    Block() : m_bitmap(), m_heap_address(), m_end_offset(), m_block_shift(), m_next_block_shift() { /* ... */ }
+                    Block() : m_bitmap(), m_heap_address(Null<KPhysicalAddress>), m_end_offset(), m_block_shift(), m_next_block_shift() { /* ... */ }
 
                     constexpr size_t GetShift() const { return m_block_shift; }
                     constexpr size_t GetNextShift() const { return m_next_block_shift; }
@@ -134,7 +134,7 @@ namespace ams::kern {
 
             void FreeBlock(KPhysicalAddress block, s32 index);
         public:
-            KPageHeap() : m_heap_address(), m_heap_size(), m_initial_used_size(), m_num_blocks(), m_blocks() { /* ... */ }
+            KPageHeap() : m_heap_address(Null<KPhysicalAddress>), m_heap_size(), m_initial_used_size(), m_num_blocks(), m_blocks() { /* ... */ }
 
             constexpr KPhysicalAddress GetAddress() const { return m_heap_address; }
             constexpr size_t GetSize() const { return m_heap_size; }

libraries/libmesosphere/include/mesosphere/kern_k_page_table_base.hpp

@@ -144,45 +144,60 @@ namespace ams::kern {
                     PageLinkedList *GetPageList() { return std::addressof(m_ll); }
             };
         private:
-            KProcessAddress m_address_space_start{};
-            KProcessAddress m_address_space_end{};
-            KProcessAddress m_heap_region_start{};
-            KProcessAddress m_heap_region_end{};
-            KProcessAddress m_current_heap_end{};
-            KProcessAddress m_alias_region_start{};
-            KProcessAddress m_alias_region_end{};
-            KProcessAddress m_stack_region_start{};
-            KProcessAddress m_stack_region_end{};
-            KProcessAddress m_kernel_map_region_start{};
-            KProcessAddress m_kernel_map_region_end{};
-            KProcessAddress m_alias_code_region_start{};
-            KProcessAddress m_alias_code_region_end{};
-            KProcessAddress m_code_region_start{};
-            KProcessAddress m_code_region_end{};
-            size_t m_max_heap_size{};
-            size_t m_mapped_physical_memory_size{};
-            size_t m_mapped_unsafe_physical_memory{};
-            size_t m_mapped_ipc_server_memory{};
-            mutable KLightLock m_general_lock{};
-            mutable KLightLock m_map_physical_memory_lock{};
-            KLightLock m_device_map_lock{};
-            KPageTableImpl m_impl{};
-            KMemoryBlockManager m_memory_block_manager{};
-            u32 m_allocate_option{};
-            u32 m_address_space_width{};
-            bool m_is_kernel{};
-            bool m_enable_aslr{};
-            bool m_enable_device_address_space_merge{};
-            KMemoryBlockSlabManager *m_memory_block_slab_manager{};
-            KBlockInfoManager *m_block_info_manager{};
-            KResourceLimit *m_resource_limit{};
-            const KMemoryRegion *m_cached_physical_linear_region{};
-            const KMemoryRegion *m_cached_physical_heap_region{};
-            MemoryFillValue m_heap_fill_value{};
-            MemoryFillValue m_ipc_fill_value{};
-            MemoryFillValue m_stack_fill_value{};
+            KProcessAddress m_address_space_start;
+            KProcessAddress m_address_space_end;
+            KProcessAddress m_heap_region_start;
+            KProcessAddress m_heap_region_end;
+            KProcessAddress m_current_heap_end;
+            KProcessAddress m_alias_region_start;
+            KProcessAddress m_alias_region_end;
+            KProcessAddress m_stack_region_start;
+            KProcessAddress m_stack_region_end;
+            KProcessAddress m_kernel_map_region_start;
+            KProcessAddress m_kernel_map_region_end;
+            KProcessAddress m_alias_code_region_start;
+            KProcessAddress m_alias_code_region_end;
+            KProcessAddress m_code_region_start;
+            KProcessAddress m_code_region_end;
+            size_t m_max_heap_size;
+            size_t m_mapped_physical_memory_size;
+            size_t m_mapped_unsafe_physical_memory;
+            size_t m_mapped_ipc_server_memory;
+            mutable KLightLock m_general_lock;
+            mutable KLightLock m_map_physical_memory_lock;
+            KLightLock m_device_map_lock;
+            KPageTableImpl m_impl;
+            KMemoryBlockManager m_memory_block_manager;
+            u32 m_allocate_option;
+            u32 m_address_space_width;
+            bool m_is_kernel;
+            bool m_enable_aslr;
+            bool m_enable_device_address_space_merge;
+            KMemoryBlockSlabManager *m_memory_block_slab_manager;
+            KBlockInfoManager *m_block_info_manager;
+            KResourceLimit *m_resource_limit;
+            const KMemoryRegion *m_cached_physical_linear_region;
+            const KMemoryRegion *m_cached_physical_heap_region;
+            MemoryFillValue m_heap_fill_value;
+            MemoryFillValue m_ipc_fill_value;
+            MemoryFillValue m_stack_fill_value;
         public:
-            constexpr KPageTableBase() { /* ... */ }
+            constexpr explicit KPageTableBase(util::ConstantInitializeTag)
+                : m_address_space_start(Null<KProcessAddress>), m_address_space_end(Null<KProcessAddress>), m_heap_region_start(Null<KProcessAddress>),
+                  m_heap_region_end(Null<KProcessAddress>), m_current_heap_end(Null<KProcessAddress>), m_alias_region_start(Null<KProcessAddress>),
+                  m_alias_region_end(Null<KProcessAddress>), m_stack_region_start(Null<KProcessAddress>), m_stack_region_end(Null<KProcessAddress>),
+                  m_kernel_map_region_start(Null<KProcessAddress>), m_kernel_map_region_end(Null<KProcessAddress>), m_alias_code_region_start(Null<KProcessAddress>),
+                  m_alias_code_region_end(Null<KProcessAddress>), m_code_region_start(Null<KProcessAddress>), m_code_region_end(Null<KProcessAddress>),
+                  m_max_heap_size(), m_mapped_physical_memory_size(), m_mapped_unsafe_physical_memory(), m_mapped_ipc_server_memory(), m_general_lock(),
+                  m_map_physical_memory_lock(), m_device_map_lock(), m_impl(util::ConstantInitialize), m_memory_block_manager(util::ConstantInitialize),
+                  m_allocate_option(), m_address_space_width(), m_is_kernel(), m_enable_aslr(), m_enable_device_address_space_merge(),
+                  m_memory_block_slab_manager(), m_block_info_manager(), m_resource_limit(), m_cached_physical_linear_region(), m_cached_physical_heap_region(),
+                  m_heap_fill_value(), m_ipc_fill_value(), m_stack_fill_value()
+            {
+                /* ... */
+            }
+
+            explicit KPageTableBase() { /* ... */ }
 
             NOINLINE Result InitializeForKernel(bool is_64_bit, void *table, KVirtualAddress start, KVirtualAddress end);
             NOINLINE Result InitializeForProcess(ams::svc::CreateProcessFlag as_type, bool enable_aslr, bool enable_device_address_space_merge, bool from_back, KMemoryManager::Pool pool, void *table, KProcessAddress start, KProcessAddress end, KProcessAddress code_address, size_t code_size, KMemoryBlockSlabManager *mem_block_slab_manager, KBlockInfoManager *block_info_manager, KResourceLimit *resource_limit);

libraries/libmesosphere/include/mesosphere/kern_k_page_table_manager.hpp

@@ -27,9 +27,10 @@ namespace ams::kern {
         private:
             using BaseHeap = KDynamicResourceManager<impl::PageTablePage, true>;
         private:
-            KPageTableSlabHeap *m_pt_heap{};
+            KPageTableSlabHeap *m_pt_heap;
         public:
-            constexpr KPageTableManager() = default;
+            constexpr explicit KPageTableManager(util::ConstantInitializeTag) : m_pt_heap() { /* ... */ }
+            explicit KPageTableManager() { /* ... */ }
 
             ALWAYS_INLINE void Initialize(KDynamicPageManager *page_allocator, KPageTableSlabHeap *pt_heap) {
                 m_pt_heap = pt_heap;

libraries/libmesosphere/include/mesosphere/kern_k_port.hpp

@@ -41,11 +41,13 @@ namespace ams::kern {
             State m_state;
             bool m_is_light;
         public:
-            constexpr KPort() : m_server(), m_client(), m_name(), m_state(State::Invalid), m_is_light() { /* ... */ }
+            explicit KPort() : m_state(State::Invalid), m_is_light() { /* ... */ }
 
             static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
             void Initialize(s32 max_sessions, bool is_light, uintptr_t name);
+            void Finalize() { /* ... */ }
+
             void OnClientClosed();
             void OnServerClosed();
 

libraries/libmesosphere/include/mesosphere/kern_k_priority_queue.hpp

@@ -235,7 +235,7 @@ namespace ams::kern {
             KPriorityQueueImpl m_suggested_queue;
         private:
             constexpr ALWAYS_INLINE void ClearAffinityBit(u64 &affinity, s32 core) {
-                affinity &= ~(u64(1ul) << core);
+                affinity &= ~(UINT64_C(1) << core);
             }
 
             constexpr ALWAYS_INLINE s32 GetNextCore(u64 &affinity) {

libraries/libmesosphere/include/mesosphere/kern_k_process.hpp

@@ -34,7 +34,7 @@
 
 namespace ams::kern {
 
-    class KProcess final : public KAutoObjectWithSlabHeapAndContainer<KProcess, KSynchronizationObject>, public KWorkerTask {
+    class KProcess final : public KAutoObjectWithSlabHeapAndContainer<KProcess, KWorkerTask> {
         MESOSPHERE_AUTOOBJECT_TRAITS(KProcess, KSynchronizationObject);
         public:
             enum State {
@@ -57,73 +57,73 @@ namespace ams::kern {
             using TLPTree = util::IntrusiveRedBlackTreeBaseTraits<KThreadLocalPage>::TreeType<KThreadLocalPage>;
             using TLPIterator = TLPTree::iterator;
         private:
-            KProcessPageTable           m_page_table{};
-            std::atomic<size_t>         m_used_kernel_memory_size{};
-            TLPTree                     m_fully_used_tlp_tree{};
-            TLPTree                     m_partially_used_tlp_tree{};
-            s32                         m_ideal_core_id{};
-            void                       *m_attached_object{};
-            KResourceLimit             *m_resource_limit{};
-            KVirtualAddress             m_system_resource_address{};
-            size_t                      m_system_resource_num_pages{};
-            size_t                      m_memory_release_hint{};
-            State                       m_state{};
-            KLightLock                  m_state_lock{};
-            KLightLock                  m_list_lock{};
-            KConditionVariable          m_cond_var{};
-            KAddressArbiter             m_address_arbiter{};
-            u64                         m_entropy[4]{};
-            bool                        m_is_signaled{};
-            bool                        m_is_initialized{};
-            bool                        m_is_application{};
-            char                        m_name[13]{};
-            std::atomic<u16>            m_num_running_threads{};
-            u32                         m_flags{};
-            KMemoryManager::Pool        m_memory_pool{};
-            s64                         m_schedule_count{};
-            KCapabilities               m_capabilities{};
-            ams::svc::ProgramId         m_program_id{};
-            u64                         m_process_id{};
-            s64                         m_creation_time{};
-            KProcessAddress             m_code_address{};
-            size_t                      m_code_size{};
-            size_t                      m_main_thread_stack_size{};
-            size_t                      m_max_process_memory{};
-            u32                         m_version{};
-            KHandleTable                m_handle_table{};
-            KProcessAddress             m_plr_address{};
-            void                       *m_plr_heap_address{};
-            KThread                    *m_exception_thread{};
-            ThreadList                  m_thread_list{};
-            SharedMemoryInfoList        m_shared_memory_list{};
-            IoRegionList                m_io_region_list{};
-            bool                        m_is_suspended{};
-            bool                        m_is_immortal{};
-            bool                        m_is_jit_debug{};
-            bool                        m_is_handle_table_initialized{};
-            ams::svc::DebugEvent        m_jit_debug_event_type{};
-            ams::svc::DebugException    m_jit_debug_exception_type{};
-            uintptr_t                   m_jit_debug_params[4]{};
-            u64                         m_jit_debug_thread_id{};
-            KWaitObject                 m_wait_object{};
-            KThread                    *m_running_threads[cpu::NumCores]{};
-            u64                         m_running_thread_idle_counts[cpu::NumCores]{};
-            KThread                    *m_pinned_threads[cpu::NumCores]{};
-            std::atomic<s64>            m_cpu_time{};
-            std::atomic<s64>            m_num_process_switches{};
-            std::atomic<s64>            m_num_thread_switches{};
-            std::atomic<s64>            m_num_fpu_switches{};
-            std::atomic<s64>            m_num_supervisor_calls{};
-            std::atomic<s64>            m_num_ipc_messages{};
-            std::atomic<s64>            m_num_ipc_replies{};
-            std::atomic<s64>            m_num_ipc_receives{};
-            KDynamicPageManager         m_dynamic_page_manager{};
-            KMemoryBlockSlabManager     m_memory_block_slab_manager{};
-            KBlockInfoManager           m_block_info_manager{};
-            KPageTableManager           m_page_table_manager{};
-            KMemoryBlockSlabHeap        m_memory_block_heap{};
-            KBlockInfoSlabHeap          m_block_info_heap{};
-            KPageTableSlabHeap          m_page_table_heap{};
+            KProcessPageTable           m_page_table;
+            util::Atomic<size_t>        m_used_kernel_memory_size;
+            TLPTree                     m_fully_used_tlp_tree;
+            TLPTree                     m_partially_used_tlp_tree;
+            s32                         m_ideal_core_id;
+            void                       *m_attached_object;
+            KResourceLimit             *m_resource_limit;
+            KVirtualAddress             m_system_resource_address;
+            size_t                      m_system_resource_num_pages;
+            size_t                      m_memory_release_hint;
+            State                       m_state;
+            KLightLock                  m_state_lock;
+            KLightLock                  m_list_lock;
+            KConditionVariable          m_cond_var;
+            KAddressArbiter             m_address_arbiter;
+            u64                         m_entropy[4];
+            bool                        m_is_signaled;
+            bool                        m_is_initialized;
+            bool                        m_is_application;
+            char                        m_name[13];
+            util::Atomic<u16>           m_num_running_threads;
+            u32                         m_flags;
+            KMemoryManager::Pool        m_memory_pool;
+            s64                         m_schedule_count;
+            KCapabilities               m_capabilities;
+            ams::svc::ProgramId         m_program_id;
+            u64                         m_process_id;
+            s64                         m_creation_time;
+            KProcessAddress             m_code_address;
+            size_t                      m_code_size;
+            size_t                      m_main_thread_stack_size;
+            size_t                      m_max_process_memory;
+            u32                         m_version;
+            KHandleTable                m_handle_table;
+            KProcessAddress             m_plr_address;
+            void                       *m_plr_heap_address;
+            KThread                    *m_exception_thread;
+            ThreadList                  m_thread_list;
+            SharedMemoryInfoList        m_shared_memory_list;
+            IoRegionList                m_io_region_list;
+            bool                        m_is_suspended;
+            bool                        m_is_immortal;
+            bool                        m_is_jit_debug;
+            bool                        m_is_handle_table_initialized;
+            ams::svc::DebugEvent        m_jit_debug_event_type;
+            ams::svc::DebugException    m_jit_debug_exception_type;
+            uintptr_t                   m_jit_debug_params[4];
+            u64                         m_jit_debug_thread_id;
+            KWaitObject                 m_wait_object;
+            KThread                    *m_running_threads[cpu::NumCores];
+            u64                         m_running_thread_idle_counts[cpu::NumCores];
+            KThread                    *m_pinned_threads[cpu::NumCores];
+            util::Atomic<s64>           m_cpu_time;
+            util::Atomic<s64>           m_num_process_switches;
+            util::Atomic<s64>           m_num_thread_switches;
+            util::Atomic<s64>           m_num_fpu_switches;
+            util::Atomic<s64>           m_num_supervisor_calls;
+            util::Atomic<s64>           m_num_ipc_messages;
+            util::Atomic<s64>           m_num_ipc_replies;
+            util::Atomic<s64>           m_num_ipc_receives;
+            KDynamicPageManager         m_dynamic_page_manager;
+            KMemoryBlockSlabManager     m_memory_block_slab_manager;
+            KBlockInfoManager           m_block_info_manager;
+            KPageTableManager           m_page_table_manager;
+            KMemoryBlockSlabHeap        m_memory_block_heap;
+            KBlockInfoSlabHeap          m_block_info_heap;
+            KPageTableSlabHeap          m_page_table_heap;
         private:
             Result Initialize(const ams::svc::CreateProcessParameter &params);
 
@@ -145,7 +145,7 @@ namespace ams::kern {
                 m_pinned_threads[core_id] = nullptr;
             }
         public:
-            KProcess() { /* ... */ }
+            explicit KProcess() : m_is_initialized(false) { /* ... */ }
 
             Result Initialize(const ams::svc::CreateProcessParameter &params, const KPageGroup &pg, const u32 *caps, s32 num_caps, KResourceLimit *res_limit, KMemoryManager::Pool pool, bool immortal);
             Result Initialize(const ams::svc::CreateProcessParameter &params, svc::KUserPointer<const u32 *> caps, s32 num_caps, KResourceLimit *res_limit, KMemoryManager::Pool pool);
@@ -284,8 +284,12 @@ namespace ams::kern {
 
             constexpr KProcessAddress GetProcessLocalRegionAddress() const { return m_plr_address; }
 
+            constexpr void *GetProcessLocalRegionHeapAddress() const { return m_plr_heap_address; }
+
+            KThread *GetExceptionThread() const { return m_exception_thread; }
+
             void AddCpuTime(s64 diff) { m_cpu_time += diff; }
-            s64 GetCpuTime() { return m_cpu_time; }
+            s64 GetCpuTime() { return m_cpu_time.Load(); }
 
             constexpr s64 GetScheduledCount() const { return m_schedule_count; }
             void IncrementScheduledCount() { ++m_schedule_count; }
@@ -380,13 +384,14 @@ namespace ams::kern {
             }
         public:
             /* Overridden parent functions. */
-            virtual bool IsInitialized() const override { return m_is_initialized; }
+            bool IsInitialized() const { return m_is_initialized; }
 
             static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
-            virtual void Finalize() override;
+            void Finalize();
 
-            virtual u64 GetId() const override final { return this->GetProcessId(); }
+            ALWAYS_INLINE u64 GetIdImpl() const { return this->GetProcessId(); }
+            ALWAYS_INLINE u64 GetId() const { return this->GetIdImpl(); }
 
             virtual bool IsSignaled() const override {
                 MESOSPHERE_ASSERT_THIS();
@@ -394,7 +399,7 @@ namespace ams::kern {
                 return m_is_signaled;
             }
 
-            virtual void DoWorkerTask() override;
+            void DoWorkerTaskImpl();
         private:
             void ChangeState(State new_state) {
                 if (m_state != new_state) {

libraries/libmesosphere/include/mesosphere/kern_k_readable_event.hpp

@@ -27,18 +27,31 @@ namespace ams::kern {
             bool m_is_signaled;
             KEvent *m_parent;
         public:
-            constexpr explicit KReadableEvent() : KSynchronizationObject(), m_is_signaled(), m_parent() { MESOSPHERE_ASSERT_THIS(); }
+            constexpr explicit KReadableEvent(util::ConstantInitializeTag) : KSynchronizationObject(util::ConstantInitialize), m_is_signaled(), m_parent() { MESOSPHERE_ASSERT_THIS(); }
+
+            explicit KReadableEvent() { /* ... */ }
 
             void Initialize(KEvent *parent);
 
             constexpr KEvent *GetParent() const { return m_parent; }
 
             Result Signal();
-            Result Clear();
+            Result Reset();
+
+            Result Clear() {
+                MESOSPHERE_ASSERT_THIS();
+
+                /* Try to perform a reset, succeeding unconditionally. */
+                this->Reset();
+
+                return ResultSuccess();
+            }
 
             virtual bool IsSignaled() const override;
             virtual void Destroy() override;
-            virtual Result Reset();
+
+            /* NOTE: This is a virtual function in Nintendo's kernel. */
+            /* virtual Result Reset(); */
     };
 
 }

libraries/libmesosphere/include/mesosphere/kern_k_resource_limit.hpp

@@ -33,12 +33,20 @@ namespace ams::kern {
             s32 m_waiter_count;
             KLightConditionVariable m_cond_var;
         public:
-            constexpr ALWAYS_INLINE KResourceLimit() : m_limit_values(), m_current_values(), m_current_hints(), m_peak_values(), m_lock(), m_waiter_count(), m_cond_var() { /* ... */ }
+            constexpr explicit ALWAYS_INLINE KResourceLimit(util::ConstantInitializeTag)
+                : KAutoObjectWithSlabHeapAndContainer<KResourceLimit, KAutoObjectWithList>(util::ConstantInitialize),
+                  m_limit_values(), m_current_values(), m_current_hints(), m_peak_values(), m_lock(), m_waiter_count(),
+                  m_cond_var(util::ConstantInitialize)
+            {
+                /* ... */
+            }
+
+            explicit ALWAYS_INLINE KResourceLimit() { /* ... */ }
 
             static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
             void Initialize();
-            virtual void Finalize() override;
+            void Finalize();
 
             s64 GetLimitValue(ams::svc::LimitableResource which) const;
             s64 GetCurrentValue(ams::svc::LimitableResource which) const;

libraries/libmesosphere/include/mesosphere/kern_k_scheduler.hpp

@@ -39,14 +39,16 @@ namespace ams::kern {
             static_assert(ams::svc::HighestThreadPriority <= HighestCoreMigrationAllowedPriority);
 
             struct SchedulingState {
-                std::atomic<u8> needs_scheduling;
-                bool interrupt_task_runnable;
-                bool should_count_idle;
-                u64  idle_count;
-                KThread *highest_priority_thread;
-                void *idle_thread_stack;
-                KThread *prev_thread;
-                KInterruptTaskManager *interrupt_task_manager;
+                util::Atomic<bool> needs_scheduling{false};
+                bool interrupt_task_runnable{false};
+                bool should_count_idle{false};
+                u64  idle_count{0};
+                KThread *highest_priority_thread{nullptr};
+                void *idle_thread_stack{nullptr};
+                KThread *prev_thread{nullptr};
+                KInterruptTaskManager *interrupt_task_manager{nullptr};
+
+                constexpr SchedulingState() = default;
             };
         private:
             friend class KScopedSchedulerLock;
@@ -58,10 +60,9 @@ namespace ams::kern {
             s32 m_core_id;
             s64 m_last_context_switch_time;
             KThread *m_idle_thread;
-            std::atomic<KThread *> m_current_thread;
+            util::Atomic<KThread *> m_current_thread;
         public:
-            constexpr KScheduler()
-                : m_state(), m_is_active(false), m_core_id(0), m_last_context_switch_time(0), m_idle_thread(nullptr), m_current_thread(nullptr)
+            constexpr KScheduler() : m_state(), m_is_active(false), m_core_id(0), m_last_context_switch_time(0), m_idle_thread(nullptr), m_current_thread(nullptr)
             {
                 m_state.needs_scheduling        = true;
                 m_state.interrupt_task_runnable = false;
@@ -102,7 +103,7 @@ namespace ams::kern {
             }
 
             ALWAYS_INLINE KThread *GetSchedulerCurrentThread() const {
-                return m_current_thread;
+                return m_current_thread.Load();
             }
 
             ALWAYS_INLINE s64 GetLastContextSwitchTime() const {
@@ -182,7 +183,7 @@ namespace ams::kern {
 
                 GetCurrentThread().EnableDispatch();
 
-                if (m_state.needs_scheduling.load()) {
+                if (m_state.needs_scheduling.Load()) {
                     /* Disable interrupts, and then check again if rescheduling is needed. */
                     KScopedInterruptDisable intr_disable;
 
@@ -192,7 +193,7 @@ namespace ams::kern {
 
             ALWAYS_INLINE void RescheduleCurrentCoreImpl() {
                 /* Check that scheduling is needed. */
-                if (AMS_LIKELY(m_state.needs_scheduling.load())) {
+                if (AMS_LIKELY(m_state.needs_scheduling.Load())) {
                     GetCurrentThread().DisableDispatch();
                     this->Schedule();
                     GetCurrentThread().EnableDispatch();
@@ -210,18 +211,6 @@ namespace ams::kern {
             static consteval bool ValidateAssemblyOffsets();
     };
 
-    consteval bool KScheduler::ValidateAssemblyOffsets() {
-        static_assert(__builtin_offsetof(KScheduler, m_state.needs_scheduling)        == KSCHEDULER_NEEDS_SCHEDULING);
-        static_assert(__builtin_offsetof(KScheduler, m_state.interrupt_task_runnable) == KSCHEDULER_INTERRUPT_TASK_RUNNABLE);
-        static_assert(__builtin_offsetof(KScheduler, m_state.highest_priority_thread) == KSCHEDULER_HIGHEST_PRIORITY_THREAD);
-        static_assert(__builtin_offsetof(KScheduler, m_state.idle_thread_stack)       == KSCHEDULER_IDLE_THREAD_STACK);
-        static_assert(__builtin_offsetof(KScheduler, m_state.prev_thread)             == KSCHEDULER_PREVIOUS_THREAD);
-        static_assert(__builtin_offsetof(KScheduler, m_state.interrupt_task_manager)  == KSCHEDULER_INTERRUPT_TASK_MANAGER);
-
-        return true;
-    }
-    static_assert(KScheduler::ValidateAssemblyOffsets());
-
     class KScopedSchedulerLock : KScopedLock<KScheduler::LockType> {
         public:
             explicit ALWAYS_INLINE KScopedSchedulerLock() : KScopedLock(KScheduler::s_scheduler_lock) { /* ... */ }

libraries/libmesosphere/include/mesosphere/kern_k_scheduler_impls.hpp

@@ -0,0 +1,43 @@
+/*
+ * Copyright (c) Atmosphère-NX
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#pragma once
+#include <mesosphere/kern_common.hpp>
+#include <mesosphere/kern_k_scheduler.hpp>
+#include <mesosphere/kern_select_interrupt_manager.hpp>
+
+namespace ams::kern {
+
+    /* NOTE: This header is included after all main headers. */
+    consteval bool KScheduler::ValidateAssemblyOffsets() {
+        static_assert(AMS_OFFSETOF(KScheduler, m_state.needs_scheduling)        == KSCHEDULER_NEEDS_SCHEDULING);
+        static_assert(AMS_OFFSETOF(KScheduler, m_state.interrupt_task_runnable) == KSCHEDULER_INTERRUPT_TASK_RUNNABLE);
+        static_assert(AMS_OFFSETOF(KScheduler, m_state.highest_priority_thread) == KSCHEDULER_HIGHEST_PRIORITY_THREAD);
+        static_assert(AMS_OFFSETOF(KScheduler, m_state.idle_thread_stack)       == KSCHEDULER_IDLE_THREAD_STACK);
+        static_assert(AMS_OFFSETOF(KScheduler, m_state.prev_thread)             == KSCHEDULER_PREVIOUS_THREAD);
+        static_assert(AMS_OFFSETOF(KScheduler, m_state.interrupt_task_manager)  == KSCHEDULER_INTERRUPT_TASK_MANAGER);
+
+        return true;
+    }
+    static_assert(KScheduler::ValidateAssemblyOffsets());
+
+    ALWAYS_INLINE void KScheduler::RescheduleOtherCores(u64 cores_needing_scheduling) {
+        if (const u64 core_mask = cores_needing_scheduling & ~(1ul << m_core_id); core_mask != 0) {
+            cpu::DataSynchronizationBarrier();
+            Kernel::GetInterruptManager().SendInterProcessorInterrupt(KInterruptName_Scheduler, core_mask);
+        }
+    }
+
+}

libraries/libmesosphere/include/mesosphere/kern_k_scheduler_lock.hpp

@@ -74,6 +74,9 @@ namespace ams::kern {
 
                 /* Release an instance of the lock. */
                 if ((--m_lock_count) == 0) {
+                    /* Perform a memory barrier here. */
+                    cpu::DataMemoryBarrierInnerShareable();
+
                     /* We're no longer going to hold the lock. Take note of what cores need scheduling. */
                     const u64 cores_needing_scheduling = SchedulerType::UpdateHighestPriorityThreads();
 

libraries/libmesosphere/include/mesosphere/kern_k_server_port.hpp

@@ -34,7 +34,8 @@ namespace ams::kern {
             LightSessionList m_light_session_list;
             KPort *m_parent;
         public:
-            constexpr KServerPort() : m_session_list(), m_light_session_list(), m_parent() { /* ... */ }
+            constexpr explicit KServerPort(util::ConstantInitializeTag) : KSynchronizationObject(util::ConstantInitialize), m_session_list(), m_light_session_list(), m_parent() { /* ... */ }
+            explicit KServerPort() { /* ... */ }
 
             void Initialize(KPort *parent);
             void EnqueueSession(KServerSession *session);

libraries/libmesosphere/include/mesosphere/kern_k_server_session.hpp

@@ -33,7 +33,8 @@ namespace ams::kern {
             KSessionRequest *m_current_request;
             KLightLock m_lock;
         public:
-            constexpr KServerSession() : m_parent(), m_request_list(), m_current_request(), m_lock() { /* ... */ }
+            constexpr explicit KServerSession(util::ConstantInitializeTag) : KSynchronizationObject(util::ConstantInitialize), m_parent(), m_request_list(), m_current_request(), m_lock() { /* ... */ }
+            explicit KServerSession() : m_current_request(nullptr), m_lock() { /* ... */ }
 
             virtual void Destroy() override;
 

libraries/libmesosphere/include/mesosphere/kern_k_session.hpp

@@ -35,33 +35,37 @@ namespace ams::kern {
                 ServerClosed = 3,
             };
         private:
+            util::Atomic<std::underlying_type<State>::type> m_atomic_state;
+            bool m_initialized;
             KServerSession m_server;
             KClientSession m_client;
-            std::atomic<std::underlying_type<State>::type> m_atomic_state;
             KClientPort *m_port;
             uintptr_t m_name;
             KProcess *m_process;
-            bool m_initialized;
         private:
             ALWAYS_INLINE void SetState(State state) {
                 m_atomic_state = static_cast<u8>(state);
             }
 
             ALWAYS_INLINE State GetState() const {
-                return static_cast<State>(m_atomic_state.load());
+                return static_cast<State>(m_atomic_state.Load());
             }
         public:
-            constexpr KSession()
-                : m_server(), m_client(), m_atomic_state(static_cast<std::underlying_type<State>::type>(State::Invalid)), m_port(), m_name(), m_process(), m_initialized()
+            constexpr explicit KSession(util::ConstantInitializeTag)
+                : KAutoObjectWithSlabHeapAndContainer<KSession, KAutoObjectWithList, true>(util::ConstantInitialize),
+                  m_atomic_state(static_cast<std::underlying_type<State>::type>(State::Invalid)), m_initialized(),
+                  m_server(util::ConstantInitialize), m_client(util::ConstantInitialize),  m_port(), m_name(), m_process()
             {
                 /* ... */
             }
 
-            void Initialize(KClientPort *client_port, uintptr_t name);
-            virtual void Finalize() override;
+            explicit KSession() : m_atomic_state(util::ToUnderlying(State::Invalid)), m_initialized(false), m_process(nullptr) { /* ... */ }
 
-            virtual bool IsInitialized() const override { return m_initialized; }
-            virtual uintptr_t GetPostDestroyArgument() const override { return reinterpret_cast<uintptr_t>(m_process); }
+            void Initialize(KClientPort *client_port, uintptr_t name);
+            void Finalize();
+
+            bool IsInitialized() const { return m_initialized; }
+            uintptr_t GetPostDestroyArgument() const { return reinterpret_cast<uintptr_t>(m_process); }
 
             static void PostDestroy(uintptr_t arg);
 

libraries/libmesosphere/include/mesosphere/kern_k_session_request.hpp

@@ -33,14 +33,14 @@ namespace ams::kern {
 
                     class Mapping {
                         private:
-                            KProcessAddress m_client_address;
-                            KProcessAddress m_server_address;
+                            uintptr_t m_client_address;
+                            uintptr_t m_server_address;
                             size_t m_size;
                             KMemoryState m_state;
                         public:
                             constexpr void Set(KProcessAddress c, KProcessAddress s, size_t sz, KMemoryState st) {
-                                m_client_address = c;
-                                m_server_address = s;
+                                m_client_address = GetInteger(c);
+                                m_server_address = GetInteger(s);
                                 m_size           = sz;
                                 m_state          = st;
                             }
@@ -57,7 +57,9 @@ namespace ams::kern {
                     u8 m_num_recv;
                     u8 m_num_exch;
                 public:
-                    constexpr explicit SessionMappings() : m_static_mappings(), m_mappings(), m_num_send(), m_num_recv(), m_num_exch() { /* ... */ }
+                    constexpr explicit SessionMappings(util::ConstantInitializeTag) : m_static_mappings(), m_mappings(), m_num_send(), m_num_recv(), m_num_exch() { /* ... */ }
+
+                    explicit SessionMappings() : m_mappings(nullptr), m_num_send(), m_num_recv(), m_num_exch() { /* ... */ }
 
                     void Initialize() { /* ... */ }
                     void Finalize();
@@ -119,8 +121,6 @@ namespace ams::kern {
                             return m_mappings[index - NumStaticMappings];
                         }
                     }
-
-
             };
         private:
             SessionMappings m_mappings;
@@ -130,20 +130,22 @@ namespace ams::kern {
             uintptr_t m_address;
             size_t m_size;
         public:
-            constexpr KSessionRequest() : m_mappings(), m_thread(), m_server(), m_event(), m_address(), m_size() { /* ... */ }
+            constexpr explicit KSessionRequest(util::ConstantInitializeTag) : KAutoObject(util::ConstantInitialize), m_mappings(util::ConstantInitialize), m_thread(), m_server(), m_event(), m_address(), m_size() { /* ... */ }
+
+            explicit KSessionRequest() : m_thread(nullptr), m_server(nullptr), m_event(nullptr) { /* ... */ }
 
             static KSessionRequest *Create() {
                 KSessionRequest *req = KSessionRequest::Allocate();
-                if (req != nullptr) {
-                    KAutoObject::Create(req);
+                if (AMS_LIKELY(req != nullptr)) {
+                    KAutoObject::Create<KSessionRequest>(req);
                 }
                 return req;
             }
 
             static KSessionRequest *CreateFromUnusedSlabMemory() {
                 KSessionRequest *req = KSessionRequest::AllocateFromUnusedSlabMemory();
-                if (req != nullptr) {
-                    KAutoObject::Create(req);
+                if (AMS_LIKELY(req != nullptr)) {
+                    KAutoObject::Create<KSessionRequest>(req);
                 }
                 return req;
             }
@@ -167,20 +169,6 @@ namespace ams::kern {
                 }
             }
 
-            virtual void Finalize() override {
-                m_mappings.Finalize();
-
-                if (m_thread) {
-                    m_thread->Close();
-                }
-                if (m_event) {
-                    m_event->Close();
-                }
-                if (m_server) {
-                    m_server->Close();
-                }
-            }
-
             static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
             constexpr ALWAYS_INLINE KThread *GetThread() const { return m_thread; }
@@ -227,6 +215,21 @@ namespace ams::kern {
             constexpr ALWAYS_INLINE KProcessAddress GetExchangeServerAddress(size_t i) const { return m_mappings.GetExchangeServerAddress(i);  }
             constexpr ALWAYS_INLINE size_t          GetExchangeSize(size_t i)          const { return m_mappings.GetExchangeSize(i);           }
             constexpr ALWAYS_INLINE KMemoryState    GetExchangeMemoryState(size_t i)   const { return m_mappings.GetExchangeMemoryState(i);    }
+        private:
+            /* NOTE: This is public and virtual in Nintendo's kernel. */
+            void Finalize() {
+                m_mappings.Finalize();
+
+                if (m_thread) {
+                    m_thread->Close();
+                }
+                if (m_event) {
+                    m_event->Close();
+                }
+                if (m_server) {
+                    m_server->Close();
+                }
+            }
     };
 
 }

libraries/libmesosphere/include/mesosphere/kern_k_shared_memory.hpp

@@ -42,9 +42,9 @@ namespace ams::kern {
             }
 
             Result Initialize(KProcess *owner, size_t size, ams::svc::MemoryPermission own_perm, ams::svc::MemoryPermission rem_perm);
-            virtual void Finalize() override;
+            void Finalize();
 
-            virtual bool IsInitialized() const override { return m_is_initialized; }
+            bool IsInitialized() const { return m_is_initialized; }
             static void PostDestroy(uintptr_t arg) { MESOSPHERE_UNUSED(arg); /* ... */ }
 
             Result Map(KProcessPageTable *table, KProcessAddress address, size_t size, KProcess *process, ams::svc::MemoryPermission map_perm);

libraries/libmesosphere/include/mesosphere/kern_k_shared_memory_info.hpp

@@ -26,7 +26,7 @@ namespace ams::kern {
             KSharedMemory *m_shared_memory;
             size_t m_reference_count;
         public:
-            constexpr KSharedMemoryInfo() : m_shared_memory(), m_reference_count() { /* ... */ }
+            explicit KSharedMemoryInfo() { /* ... */ }
             ~KSharedMemoryInfo() { /* ... */ }
 
             constexpr void Initialize(KSharedMemory *m) {