Implement the NCM sysmodule (closes #91)

* Implement NCM * Modernize ncm_main * Remove unnecessary smExit * Give access to svcCallSecureMonitor * Stack size bump * Fix incorrect setup for NandUser's content storage entry * Fix a potential data abort when flushing the placeholder accessor cache * Fix HasFile and HasDirectory * Use r+b, not w+b * Misc fixes * errno begone * Fixed more stdio error handling * More main fixes * Various command improvements * Make dispatch tables great again * Fix logic inversion * Fixed content path generation * Bump heap size, fix CleanupAllPlaceHolder * Various fixes. Note: This contains debug stuff which will be removed later. I was getting tired of having to cherrypick tiny changes * Fixed placeholder/content deletion * Fixed incorrect content manager destruction * Prevent automatic placeholder creation on open * Fixed List implementation. Also lots of debug logging. * Removed debug code * Added a scope guard for WritePlaceHolder * Manually prevent placeholder/content appending * Revert "Removed debug code" This reverts commit d6ff261fcc. * Always cache placeholder file. Switch to ftell for preventing appending * Universally use EnsureEnabled * Abstract away file writing logic * Misc cleanup * Refactor placeholder cacheing * Remove debug code (again) * Revert "Remove debug code (again)" This reverts commit 168447d80e. * Misc changes * Fixed file modes * Fixed ContentId/PlaceHolderId alignment * Improved type safety * Fixed reinitialization * Fixed doubleup on path creation * Remove debug code * Fixed 1.0.0 booting * Correct amount of add on content * Correct main thread stack size * lr: Introducing registered data * Reorder stratosphere Makefile * Move results to libstrat * lr: Cleanup lr_redirection * lr: lr_manager tweaks * lr: Imrpoved path handling and adjust ResolveAddOnContentPath order * lr: Organise types * Add eof newlines * lr: Eliminate unnecessary vars * lr: Unnecessary vars 2 electric boogaloo * lr: Various helpers * lr: RegisteredLocationResolver helpers * ncm: Move ncm_types to libstrat * ncm: Misc cleanup * Implement NCM * Modernize ncm_main * Remove unnecessary smExit * Give access to svcCallSecureMonitor * Stack size bump * Fix incorrect setup for NandUser's content storage entry * Fix a potential data abort when flushing the placeholder accessor cache * Fix HasFile and HasDirectory * Use r+b, not w+b * Misc fixes * errno begone * Fixed more stdio error handling * More main fixes * Various command improvements * Make dispatch tables great again * Fix logic inversion * Fixed content path generation * Bump heap size, fix CleanupAllPlaceHolder * Various fixes. Note: This contains debug stuff which will be removed later. I was getting tired of having to cherrypick tiny changes * Fixed placeholder/content deletion * Fixed incorrect content manager destruction * Prevent automatic placeholder creation on open * Fixed List implementation. Also lots of debug logging. * Removed debug code * Added a scope guard for WritePlaceHolder * Manually prevent placeholder/content appending * Revert "Removed debug code" This reverts commit d6ff261fcc. * Always cache placeholder file. Switch to ftell for preventing appending * Universally use EnsureEnabled * Abstract away file writing logic * Misc cleanup * Refactor placeholder cacheing * Remove debug code (again) * Revert "Remove debug code (again)" This reverts commit 168447d80e. * Misc changes * Fixed file modes * Fixed ContentId/PlaceHolderId alignment * Improved type safety * Fixed reinitialization * Fixed doubleup on path creation * Remove debug code * Fixed 1.0.0 booting * Correct amount of add on content * Correct main thread stack size * lr: Introducing registered data * Reorder stratosphere Makefile * Move results to libstrat * lr: Cleanup lr_redirection * lr: lr_manager tweaks * lr: Imrpoved path handling and adjust ResolveAddOnContentPath order * lr: Organise types * Add eof newlines * lr: Eliminate unnecessary vars * lr: Unnecessary vars 2 electric boogaloo * lr: Various helpers * lr: RegisteredLocationResolver helpers * ncm: Move ncm_types to libstrat * ncm: Misc cleanup * Updated AddOnContentLocationResolver and RegisteredLocationResolver to 9.0.0 * Finished updating lr to 9.0.0 * Updated NCM to 9.0.0 * Fix libstrat includes * Fixed application launching * title_id_2 -> owner_tid * Updated to new-ipc * Change to using pure virtuals * Title Id -> Program Id * Fixed compilation against master * std::scoped_lock<> -> std::scoped_lock * Adopted R_UNLESS and R_CONVERT * Prefix namespace to Results * Adopt std::numeric_limits * Fixed incorrect error handling in ReadFile * Adopted AMS_ABORT_UNLESS * Adopt util::GenerateUuid() * Syntax improvements * ncm_types: Address review * Address more review comments * Updated copyrights * Address more feedback * More feedback addressed * More changes * Move dispatch tables out of interface files * Addressed remaining comments * lr: move into libstratosphere * ncm: Fix logic inversion * lr: Add comments * lr: Remove whitespace * ncm: Start addressing feedback * ncm: Cleanup InitializeContentManager * lr: support client-side usage * lr_service -> lr_api * ncm: Begin refactoring content manager * ncm: More content manager improvements * ncm: Content manager mount improvements * ldr: use lr bindings * lr bindings usage: minor fixes * ncm/lr: Pointer placement * ncm: placeholder accessor cleanup * ncm: minor fixes * ncm: refactor rights cache * ncm: content meta database cleanup * ncm: move content meta database impl out of interface file * ncm: Use const ContentMetaKey & * ncm: fix other non-const ContentMetaKey references * ncm: content meta database cleanup * ncm: content storage fixes for 2.0.0 * ncm: add missing end of file newlines * ncm: implement ContentMetaReader * ncm: client-side api * ncm: trim trailing spaces * ncm: FS_MAX_PATH-1 -> fs::EntryNameLengthMax * ncm: Use PathString and Path * fs: implement accessor wrappers for ncm * fs: implement user fs wrappers * fs: add MountSdCard * ncm: move to content manager impl * ncm: fix up main * kvdb: use fs:: * fs: Add wrappers needed for ncm * ncm: use fs bindings, other refactoring * ncm: minor fixes * fsa: fix ReadFile without size output * fs: add substorage, rom path tool * ncm: fix dangling fsdev usage * fs: fix bug in Commit * fs: fixed incorrect mode check * fs: implement Mount(System)Data * ncm: don't delete hos * results: add R_SUCCEED_IF * ams-except-ncm: use R_SUCCEED_IF * ncm: added comments * ncm: fix api definitions * ncm: use R_SUCCEED_IF * pm: think of the savings * ncm: employ kernel strats * ncm: Nintendo has 5 MiB of heap. Give ourselves 4 to be safe, pending analysis * ncm: refactor IDs, split types header into many headers * ams.mitm: use fs bindings instead of stdio * fs: SystemData uses SystemDataId * ncm: improve meta-db accuracy * ncm: inline getlatestkey * fs: improve UnsupportedOperation results * fs: modernize mount utils * ams: misc fixes for merge-errors * fs: improve unsupportedoperation results * git subrepo pull emummc subrepo: subdir: "emummc" merged: "d12dd546" upstream: origin: "https://github.com/m4xw/emuMMC" branch: "develop" commit: "d12dd546" git-subrepo: version: "0.4.1" origin: "???" commit: "???" * util: add boundedmap * ncm: minor style fixes * ncm: don't unmount if mounting fails * lr: bug fixes * ncm: implement ncm.for-initialize + ncm.for-safemode * lr: ncm::ProgramId::Invalid -> ncm::InvalidProgramId * ncm: fix open directory mode on 1.0.0 * ncm: fix fs use, implement more of < 4.0.0 for-initialize/safemode * ncm: implement packagedcontent -> content for building metadb * ncm: fix save data flag management * ncm: address some review suggestions (thanks @leoetlino!) * updater: use fs bindings * fs: implement MountCode * fs: prefer make_unique to operator new * ncm: implement remaining ContentMetaDatabaseBuilder functionality Co-authored-by: Michael Scire <SciresM@gmail.com>
2020-03-08 19:06:23 +11:00
parent f9403201f0
commit c7026b9094
254 changed files with 16876 additions and 1274 deletions
--- a/stratosphere/ams_mitm/source/amsmitm_main.cpp
+++ b/stratosphere/ams_mitm/source/amsmitm_main.cpp
@@ -40,7 +40,7 @@ extern "C" {

 namespace ams {

-    ncm::ProgramId CurrentProgramId = ncm::ProgramId::AtmosphereMitm;
+    ncm::ProgramId CurrentProgramId = ncm::AtmosphereProgramId::Mitm;

    namespace result {

--- a/stratosphere/ams_mitm/source/bpc_mitm/bpc_mitm_service.hpp
+++ b/stratosphere/ams_mitm/source/bpc_mitm/bpc_mitm_service.hpp
@@ -31,8 +31,8 @@ namespace ams::mitm::bpc {
                 * - fatal, to simplify payload reboot logic significantly
                 * - hbl, to allow homebrew to take advantage of the feature.
                 */
-                return client_info.program_id == ncm::ProgramId::Am ||
-                       client_info.program_id == ncm::ProgramId::Fatal ||
+                return client_info.program_id == ncm::SystemProgramId::Am ||
+                       client_info.program_id == ncm::SystemProgramId::Fatal ||
                       client_info.override_status.IsHbl();
            }
        public:
--- a/stratosphere/ams_mitm/source/fs_mitm/fs_mitm_service.cpp
+++ b/stratosphere/ams_mitm/source/fs_mitm/fs_mitm_service.cpp
@@ -137,7 +137,7 @@ namespace ams::mitm::fs {

        Result OpenWebContentFileSystem(sf::Out<std::shared_ptr<IFileSystemInterface>> &out, ncm::ProgramId client_program_id, ncm::ProgramId program_id, FsFileSystemType filesystem_type, Service *fwd, const fssrv::sf::Path *path, bool with_id, bool try_program_specific) {
            /* Check first that we're a web applet opening web content. */
-            R_UNLESS(ncm::IsWebAppletProgramId(client_program_id),      sm::mitm::ResultShouldForwardToSession());
+            R_UNLESS(ncm::IsWebAppletId(client_program_id),             sm::mitm::ResultShouldForwardToSession());
            R_UNLESS(filesystem_type == FsFileSystemType_ContentManual, sm::mitm::ResultShouldForwardToSession());

            /* Try to mount the HBL web filesystem. If this succeeds then we're done. */
@@ -162,8 +162,8 @@ namespace ams::mitm::fs {

    Result FsMitmService::OpenSdCardFileSystem(sf::Out<std::shared_ptr<IFileSystemInterface>> out) {
        /* We only care about redirecting this for NS/emummc. */
-        R_UNLESS(this->client_info.program_id == ncm::ProgramId::Ns, sm::mitm::ResultShouldForwardToSession());
-        R_UNLESS(emummc::IsActive(),                                 sm::mitm::ResultShouldForwardToSession());
+        R_UNLESS(this->client_info.program_id == ncm::SystemProgramId::Ns, sm::mitm::ResultShouldForwardToSession());
+        R_UNLESS(emummc::IsActive(),                                       sm::mitm::ResultShouldForwardToSession());

        /* Create a new SD card filesystem. */
        FsFileSystem sd_fs;
@@ -178,7 +178,7 @@ namespace ams::mitm::fs {

    Result FsMitmService::OpenSaveDataFileSystem(sf::Out<std::shared_ptr<IFileSystemInterface>> out, u8 _space_id, const FsSaveDataAttribute &attribute) {
        /* We only want to intercept saves for games, right now. */
-        const bool is_game_or_hbl = this->client_info.override_status.IsHbl() || ncm::IsApplicationProgramId(this->client_info.program_id);
+        const bool is_game_or_hbl = this->client_info.override_status.IsHbl() || ncm::IsApplicationId(this->client_info.program_id);
        R_UNLESS(is_game_or_hbl, sm::mitm::ResultShouldForwardToSession());

        /* Only redirect if the appropriate system setting is set. */
@@ -333,10 +333,13 @@ namespace ams::mitm::fs {
        return ResultSuccess();
    }

-    Result FsMitmService::OpenDataStorageByDataId(sf::Out<std::shared_ptr<IStorageInterface>> out, ncm::ProgramId /* TODO: ncm::DataId */ data_id, u8 storage_id) {
+    Result FsMitmService::OpenDataStorageByDataId(sf::Out<std::shared_ptr<IStorageInterface>> out, ncm::DataId _data_id, u8 storage_id) {
        /* Only mitm if we should override contents for the current process. */
        R_UNLESS(this->client_info.override_status.IsProgramSpecific(), sm::mitm::ResultShouldForwardToSession());

+        /* TODO: Decide how to handle DataId vs ProgramId for this API. */
+        const ncm::ProgramId data_id = {_data_id.value};
+
        /* Only mitm if there is actually an override romfs. */
        R_UNLESS(mitm::fs::HasSdRomfsContent(data_id),                  sm::mitm::ResultShouldForwardToSession());

--- a/stratosphere/ams_mitm/source/fs_mitm/fs_mitm_service.hpp
+++ b/stratosphere/ams_mitm/source/fs_mitm/fs_mitm_service.hpp
@@ -50,12 +50,12 @@ namespace ams::mitm::fs {
                }

                /* We want to mitm ns, to intercept SD card requests. */
-                if (program_id == ncm::ProgramId::Ns) {
+                if (program_id == ncm::SystemProgramId::Ns) {
                    return true;
                }

                /* We want to mitm sdb, to support sd-romfs redirection of common system archives (like system font, etc). */
-                if (program_id == ncm::ProgramId::Sdb) {
+                if (program_id == ncm::SystemProgramId::Sdb) {
                    return true;
                }

@@ -69,7 +69,7 @@ namespace ams::mitm::fs {
                /* Figure out why, and address it. */
                /* TODO: This may be because pre-rewrite code really mismanaged domain objects in a way that would cause bad things. */
                /* Need to verify if this is fixed now. */
-                if (client_info.program_id == ncm::ProgramId::AppletQlaunch || client_info.program_id == ncm::ProgramId::AppletMaintenanceMenu) {
+                if (client_info.program_id == ncm::SystemAppletId::Qlaunch || client_info.program_id == ncm::SystemAppletId::MaintenanceMenu) {
                    has_launched_qlaunch = true;
                }

@@ -85,7 +85,7 @@ namespace ams::mitm::fs {
            Result OpenSaveDataFileSystem(sf::Out<std::shared_ptr<IFileSystemInterface>> out, u8 space_id, const FsSaveDataAttribute &attribute);
            Result OpenBisStorage(sf::Out<std::shared_ptr<IStorageInterface>> out, u32 bis_partition_id);
            Result OpenDataStorageByCurrentProcess(sf::Out<std::shared_ptr<IStorageInterface>> out);
-            Result OpenDataStorageByDataId(sf::Out<std::shared_ptr<IStorageInterface>> out, ncm::ProgramId /* TODO: ncm::DataId */ data_id, u8 storage_id);
+            Result OpenDataStorageByDataId(sf::Out<std::shared_ptr<IStorageInterface>> out, ncm::DataId data_id, u8 storage_id);
        public:
            DEFINE_SERVICE_DISPATCH_TABLE {
                MAKE_SERVICE_COMMAND_META(OpenFileSystemWithPatch, hos::Version_200),
--- a/stratosphere/ams_mitm/source/fs_mitm/fsmitm_boot0storage.cpp
+++ b/stratosphere/ams_mitm/source/fs_mitm/fsmitm_boot0storage.cpp
@@ -31,7 +31,7 @@ namespace ams::mitm::fs {
            /* RCM bug patched. */
            /* Only allow NS to update the BCT pubks. */
            /* AutoRCM on a patched unit will cause a brick, so homebrew should NOT be allowed to write. */
-            return this->client_info.program_id == ncm::ProgramId::Ns;
+            return this->client_info.program_id == ncm::SystemProgramId::Ns;
        } else {
            /* RCM bug unpatched. */
            /* Allow homebrew but not NS to update the BCT pubks. */
--- a/stratosphere/ams_mitm/source/fs_mitm/fsmitm_boot0storage.hpp
+++ b/stratosphere/ams_mitm/source/fs_mitm/fsmitm_boot0storage.hpp
@@ -140,8 +140,7 @@ namespace ams::mitm::fs {
        private:
            bool CanModifyBctPublicKey();
        public:
-            Boot0Storage(FsStorage *s, const sm::MitmProcessInfo &c) : Base(s), client_info(c) { /* ... */ }
-            Boot0Storage(FsStorage s,  const sm::MitmProcessInfo &c) : Base(s), client_info(c) { /* ... */ }
+            Boot0Storage(FsStorage &s,  const sm::MitmProcessInfo &c) : Base(s), client_info(c) { /* ... */ }
        public:
            virtual Result Read(s64 offset, void *_buffer, size_t size) override;
            virtual Result Write(s64 offset, const void *_buffer, size_t size) override;
--- a/stratosphere/ams_mitm/source/fs_mitm/fsmitm_layered_romfs_storage.cpp
+++ b/stratosphere/ams_mitm/source/fs_mitm/fsmitm_layered_romfs_storage.cpp
@@ -98,7 +98,7 @@ namespace ams::mitm::fs {
    Result LayeredRomfsStorage::Read(s64 offset, void *buffer, size_t size) {
        /* Check if we can succeed immediately. */
        R_UNLESS(size >= 0, fs::ResultInvalidSize());
-        R_UNLESS(size > 0,  ResultSuccess());
+        R_SUCCEED_IF(size == 0);

        /* Ensure we're initialized. */
        if (!this->is_initialized) {
--- a/stratosphere/ams_mitm/source/fs_mitm/fsmitm_layered_romfs_storage.hpp
+++ b/stratosphere/ams_mitm/source/fs_mitm/fsmitm_layered_romfs_storage.hpp
@@ -49,6 +49,16 @@ namespace ams::mitm::fs {
            virtual Result GetSize(s64 *out_size) override;
            virtual Result Flush() override;
            virtual Result OperateRange(void *dst, size_t dst_size, ams::fs::OperationId op_id, s64 offset, s64 size, const void *src, size_t src_size) override;
+
+            virtual Result Write(s64 offset, const void *buffer, size_t size) override {
+                /* TODO: Better result code? */
+                return ams::fs::ResultUnsupportedOperation();
+            }
+
+            virtual Result SetSize(s64 size) override {
+                /* TODO: Better result code? */
+                return ams::fs::ResultUnsupportedOperation();
+            }
    };

 }
--- a/stratosphere/ams_mitm/source/ns_mitm/ns_am_mitm_service.hpp
+++ b/stratosphere/ams_mitm/source/ns_mitm/ns_am_mitm_service.hpp
@@ -30,7 +30,7 @@ namespace ams::mitm::ns {
                /* We will mitm:
                 * - web applets, to facilitate hbl web browser launching.
                 */
-                return ncm::IsWebAppletProgramId(client_info.program_id);
+                return ncm::IsWebAppletId(client_info.program_id);
            }
        public:
            SF_MITM_SERVICE_OBJECT_CTOR(NsAmMitmService) { /* ... */ }
--- a/stratosphere/ams_mitm/source/ns_mitm/ns_web_mitm_service.hpp
+++ b/stratosphere/ams_mitm/source/ns_mitm/ns_web_mitm_service.hpp
@@ -59,7 +59,7 @@ namespace ams::mitm::ns {
                /* We will mitm:
                 * - web applets, to facilitate hbl web browser launching.
                 */
-                return ncm::IsWebAppletProgramId(client_info.program_id);
+                return ncm::IsWebAppletId(client_info.program_id);
            }
        public:
            SF_MITM_SERVICE_OBJECT_CTOR(NsWebMitmService) { /* ... */ }
--- a/stratosphere/ams_mitm/source/set_mitm/set_mitm_service.cpp
+++ b/stratosphere/ams_mitm/source/set_mitm/set_mitm_service.cpp
@@ -22,12 +22,12 @@ namespace ams::mitm::settings {
    Result SetMitmService::EnsureLocale() {
        std::scoped_lock lk(this->lock);

-        const bool is_ns = this->client_info.program_id == ncm::ProgramId::Ns;
+        const bool is_ns = this->client_info.program_id == ncm::SystemProgramId::Ns;

        if (!this->got_locale) {
            std::memset(&this->locale, 0xCC, sizeof(this->locale));
            ncm::ProgramId program_id = this->client_info.program_id;
-            if (program_id == ncm::ProgramId::Ns) {
+            if (is_ns) {
                /* When NS asks for a locale, refresh to get the current application locale. */
                os::ProcessId application_process_id;
                R_TRY(pm::dmnt::GetApplicationProcessId(&application_process_id));
--- a/stratosphere/ams_mitm/source/set_mitm/set_mitm_service.hpp
+++ b/stratosphere/ams_mitm/source/set_mitm/set_mitm_service.hpp
@@ -33,8 +33,8 @@ namespace ams::mitm::settings {
                /* We will mitm:
                 * - ns and games, to allow for overriding game locales.
                 */
-                const bool is_game = (ncm::IsApplicationProgramId(client_info.program_id) && !client_info.override_status.IsHbl());
-                return client_info.program_id == ncm::ProgramId::Ns || is_game;
+                const bool is_game = (ncm::IsApplicationId(client_info.program_id) && !client_info.override_status.IsHbl());
+                return client_info.program_id == ncm::SystemProgramId::Ns || is_game;
            }
        public:
            SF_MITM_SERVICE_OBJECT_CTOR(SetMitmService) {
--- a/stratosphere/ams_mitm/source/set_mitm/setsys_mitm_service.cpp
+++ b/stratosphere/ams_mitm/source/set_mitm/setsys_mitm_service.cpp
@@ -35,17 +35,17 @@ namespace ams::mitm::settings {
            }

            /* Mount firmware version data archive. */
-            R_ABORT_UNLESS(romfsMountFromDataArchive(static_cast<u64>(ncm::ProgramId::ArchiveSystemVersion), NcmStorageId_BuiltInSystem, "sysver"));
            {
-                ON_SCOPE_EXIT { romfsUnmount("sysver"); };
+                R_ABORT_UNLESS(ams::fs::MountSystemData("sysver", ncm::SystemDataId::SystemVersion));
+                ON_SCOPE_EXIT { ams::fs::Unmount("sysver"); };

                /* Firmware version file must exist. */
-                FILE *fp = fopen("sysver:/file", "rb");
-                AMS_ABORT_UNLESS(fp != nullptr);
-                ON_SCOPE_EXIT { fclose(fp); };
+                ams::fs::FileHandle file;
+                R_ABORT_UNLESS(ams::fs::OpenFile(std::addressof(file), "sysver:/file", fs::OpenMode_Read));
+                ON_SCOPE_EXIT { ams::fs::CloseFile(file); };

                /* Must be possible to read firmware version from file. */
-                AMS_ABORT_UNLESS(fread(&g_firmware_version, sizeof(g_firmware_version), 1, fp) == 1);
+                R_ABORT_UNLESS(ams::fs::ReadFile(file, 0, std::addressof(g_firmware_version), sizeof(g_firmware_version)));

                g_ams_firmware_version = g_firmware_version;
            }
@@ -75,7 +75,7 @@ namespace ams::mitm::settings {

            /* We want to give a special firmware version to the home menu title, and nothing else. */
            /* This means Qlaunch + Maintenance Menu, and nothing else. */
-            if (client_info.program_id == ncm::ProgramId::AppletQlaunch || client_info.program_id == ncm::ProgramId::AppletMaintenanceMenu) {
+            if (client_info.program_id == ncm::SystemAppletId::Qlaunch || client_info.program_id == ncm::SystemAppletId::MaintenanceMenu) {
                *out = g_ams_firmware_version;
            } else {
                *out = g_firmware_version;