OmniNX/Atmosphere
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement the NCM sysmodule (closes #91)

Browse Source 
* Implement NCM

* Modernize ncm_main

* Remove unnecessary smExit

* Give access to svcCallSecureMonitor

* Stack size bump

* Fix incorrect setup for NandUser's content storage entry

* Fix a potential data abort when flushing the placeholder accessor cache

* Fix HasFile and HasDirectory

* Use r+b, not w+b

* Misc fixes

* errno begone

* Fixed more stdio error handling

* More main fixes

* Various command improvements

* Make dispatch tables great again

* Fix logic inversion

* Fixed content path generation

* Bump heap size, fix CleanupAllPlaceHolder

* Various fixes. Note: This contains debug stuff which will be removed later. I was getting tired of having to cherrypick tiny changes

* Fixed placeholder/content deletion

* Fixed incorrect content manager destruction

* Prevent automatic placeholder creation on open

* Fixed List implementation. Also lots of debug logging.

* Removed debug code

* Added a scope guard for WritePlaceHolder

* Manually prevent placeholder/content appending

* Revert "Removed debug code"

This reverts commit d6ff261fcc.

* Always cache placeholder file. Switch to ftell for preventing appending

* Universally use EnsureEnabled

* Abstract away file writing logic

* Misc cleanup

* Refactor placeholder cacheing

* Remove debug code (again)

* Revert "Remove debug code (again)"

This reverts commit 168447d80e.

* Misc changes

* Fixed file modes

* Fixed ContentId/PlaceHolderId alignment

* Improved type safety

* Fixed reinitialization

* Fixed doubleup on path creation

* Remove debug code

* Fixed 1.0.0 booting

* Correct amount of add on content

* Correct main thread stack size

* lr: Introducing registered data

* Reorder stratosphere Makefile

* Move results to libstrat

* lr: Cleanup lr_redirection

* lr: lr_manager tweaks

* lr: Imrpoved path handling and adjust ResolveAddOnContentPath order

* lr: Organise types

* Add eof newlines

* lr: Eliminate unnecessary vars

* lr: Unnecessary vars 2 electric boogaloo

* lr: Various helpers

* lr: RegisteredLocationResolver helpers

* ncm: Move ncm_types to libstrat

* ncm: Misc cleanup

* Implement NCM

* Modernize ncm_main

* Remove unnecessary smExit

* Give access to svcCallSecureMonitor

* Stack size bump

* Fix incorrect setup for NandUser's content storage entry

* Fix a potential data abort when flushing the placeholder accessor cache

* Fix HasFile and HasDirectory

* Use r+b, not w+b

* Misc fixes

* errno begone

* Fixed more stdio error handling

* More main fixes

* Various command improvements

* Make dispatch tables great again

* Fix logic inversion

* Fixed content path generation

* Bump heap size, fix CleanupAllPlaceHolder

* Various fixes. Note: This contains debug stuff which will be removed later. I was getting tired of having to cherrypick tiny changes

* Fixed placeholder/content deletion

* Fixed incorrect content manager destruction

* Prevent automatic placeholder creation on open

* Fixed List implementation. Also lots of debug logging.

* Removed debug code

* Added a scope guard for WritePlaceHolder

* Manually prevent placeholder/content appending

* Revert "Removed debug code"

This reverts commit d6ff261fcc.

* Always cache placeholder file. Switch to ftell for preventing appending

* Universally use EnsureEnabled

* Abstract away file writing logic

* Misc cleanup

* Refactor placeholder cacheing

* Remove debug code (again)

* Revert "Remove debug code (again)"

This reverts commit 168447d80e.

* Misc changes

* Fixed file modes

* Fixed ContentId/PlaceHolderId alignment

* Improved type safety

* Fixed reinitialization

* Fixed doubleup on path creation

* Remove debug code

* Fixed 1.0.0 booting

* Correct amount of add on content

* Correct main thread stack size

* lr: Introducing registered data

* Reorder stratosphere Makefile

* Move results to libstrat

* lr: Cleanup lr_redirection

* lr: lr_manager tweaks

* lr: Imrpoved path handling and adjust ResolveAddOnContentPath order

* lr: Organise types

* Add eof newlines

* lr: Eliminate unnecessary vars

* lr: Unnecessary vars 2 electric boogaloo

* lr: Various helpers

* lr: RegisteredLocationResolver helpers

* ncm: Move ncm_types to libstrat

* ncm: Misc cleanup

* Updated AddOnContentLocationResolver and RegisteredLocationResolver to 9.0.0

* Finished updating lr to 9.0.0

* Updated NCM to 9.0.0

* Fix libstrat includes

* Fixed application launching

* title_id_2 -> owner_tid

* Updated to new-ipc

* Change to using pure virtuals

* Title Id -> Program Id

* Fixed compilation against master

* std::scoped_lock<> -> std::scoped_lock

* Adopted R_UNLESS and R_CONVERT

* Prefix namespace to Results

* Adopt std::numeric_limits

* Fixed incorrect error handling in ReadFile

* Adopted AMS_ABORT_UNLESS

* Adopt util::GenerateUuid()

* Syntax improvements

* ncm_types: Address review

* Address more review comments

* Updated copyrights

* Address more feedback

* More feedback addressed

* More changes

* Move dispatch tables out of interface files

* Addressed remaining comments

* lr: move into libstratosphere

* ncm: Fix logic inversion

* lr: Add comments

* lr: Remove whitespace

* ncm: Start addressing feedback

* ncm: Cleanup InitializeContentManager

* lr: support client-side usage

* lr_service -> lr_api

* ncm: Begin refactoring content manager

* ncm: More content manager improvements

* ncm: Content manager mount improvements

* ldr: use lr bindings

* lr bindings usage: minor fixes

* ncm/lr: Pointer placement

* ncm: placeholder accessor cleanup

* ncm: minor fixes

* ncm: refactor rights cache

* ncm: content meta database cleanup

* ncm: move content meta database impl out of interface file

* ncm: Use const ContentMetaKey &

* ncm: fix other non-const ContentMetaKey references

* ncm: content meta database cleanup

* ncm: content storage fixes for 2.0.0

* ncm: add missing end of file newlines

* ncm: implement ContentMetaReader

* ncm: client-side api

* ncm: trim trailing spaces

* ncm: FS_MAX_PATH-1 -> fs::EntryNameLengthMax

* ncm: Use PathString and Path

* fs: implement accessor wrappers for ncm

* fs: implement user fs wrappers

* fs: add MountSdCard

* ncm: move to content manager impl

* ncm: fix up main

* kvdb: use fs::

* fs: Add wrappers needed for ncm

* ncm: use fs bindings, other refactoring

* ncm: minor fixes

* fsa: fix ReadFile without size output

* fs: add substorage, rom path tool

* ncm: fix dangling fsdev usage

* fs: fix bug in Commit

* fs: fixed incorrect mode check

* fs: implement Mount(System)Data

* ncm: don't delete hos

* results: add R_SUCCEED_IF

* ams-except-ncm: use R_SUCCEED_IF

* ncm: added comments

* ncm: fix api definitions

* ncm: use R_SUCCEED_IF

* pm: think of the savings

* ncm: employ kernel strats

* ncm: Nintendo has 5 MiB of heap. Give ourselves 4 to be safe, pending analysis

* ncm: refactor IDs, split types header into many headers

* ams.mitm: use fs bindings instead of stdio

* fs: SystemData uses SystemDataId

* ncm: improve meta-db accuracy

* ncm: inline getlatestkey

* fs: improve UnsupportedOperation results

* fs: modernize mount utils

* ams: misc fixes for merge-errors

* fs: improve unsupportedoperation results

* git subrepo pull emummc

subrepo:
  subdir:   "emummc"
  merged:   "d12dd546"
upstream:
  origin:   "https://github.com/m4xw/emuMMC"
  branch:   "develop"
  commit:   "d12dd546"
git-subrepo:
  version:  "0.4.1"
  origin:   "???"
  commit:   "???"

* util: add boundedmap

* ncm: minor style fixes

* ncm: don't unmount if mounting fails

* lr: bug fixes

* ncm: implement ncm.for-initialize + ncm.for-safemode

* lr: ncm::ProgramId::Invalid -> ncm::InvalidProgramId

* ncm: fix open directory mode on 1.0.0

* ncm: fix fs use, implement more of < 4.0.0 for-initialize/safemode

* ncm: implement packagedcontent -> content for building metadb

* ncm: fix save data flag management

* ncm: address some review suggestions (thanks @leoetlino!)

* updater: use fs bindings

* fs: implement MountCode

* fs: prefer make_unique to operator new

* ncm: implement remaining ContentMetaDatabaseBuilder functionality

Co-authored-by: Michael Scire <SciresM@gmail.com>
This commit is contained in:
Adubbz
2020-03-08 19:06:23 +11:00
committed by GitHub
parent f9403201f0
commit c7026b9094
254 changed files with 16876 additions and 1274 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
libraries/libstratosphere/include/stratosphere/kvdb/kvdb_archive.hpp
View File

@@ -13,9 +13,8 @@
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#pragma once
#include "kvdb_auto_buffer.hpp"
#include <stratosphere/kvdb/kvdb_auto_buffer.hpp>
namespace ams::kvdb {

8
libraries/libstratosphere/include/stratosphere/kvdb/kvdb_auto_buffer.hpp
View File

@@ -13,7 +13,6 @@
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#pragma once
#include <vapours.hpp>
@@ -39,7 +38,7 @@ namespace ams::kvdb {
}
AutoBuffer& operator=(AutoBuffer &&rhs) {
rhs.Swap(*this);
AutoBuffer(std::move(rhs)).Swap(*this);
return *this;
}
@@ -70,9 +69,8 @@ namespace ams::kvdb {
/* Allocate a buffer. */
this->buffer = static_cast<u8 *>(std::malloc(size));
if (this->buffer == nullptr) {
return ResultAllocationFailed();
}
R_UNLESS(this->buffer != nullptr, ResultAllocationFailed());
this->size = size;
return ResultSuccess();
}

3
libraries/libstratosphere/include/stratosphere/kvdb/kvdb_bounded_string.hpp
View File

@@ -13,7 +13,6 @@
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#pragma once
#include <vapours.hpp>
@@ -51,6 +50,7 @@ namespace ams::kvdb {
std::va_list args;
va_start(args, format);
CheckLength(std::vsnprintf(string.buffer, N, format, args));
string.buffer[N - 1] = 0;
va_end(args);
return string;
@@ -74,6 +74,7 @@ namespace ams::kvdb {
/* Ensure string can fit in our buffer. */
CheckLength(strnlen(s, N));
std::strncpy(this->buffer, s, N);
this->buffer[N - 1] = 0;
}
void SetFormat(const char *format, ...) __attribute__((format (printf, 2, 3))) {

75
libraries/libstratosphere/include/stratosphere/kvdb/kvdb_file_key_value_cache.hpp
View File

@@ -13,9 +13,10 @@
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <sys/stat.h>
#include "kvdb_bounded_string.hpp"
#include "kvdb_file_key_value_store.hpp"
#include <stratosphere/fs/fs_filesystem.hpp>
#include <stratosphere/fs/fs_file.hpp>
#include <stratosphere/kvdb/kvdb_bounded_string.hpp>
#include <stratosphere/kvdb/kvdb_file_key_value_store.hpp>
namespace ams::kvdb {
@@ -39,16 +40,16 @@ namespace ams::kvdb {
public:
static Result CreateNewList(const char *path) {
/* Create new lru_list.dat. */
R_TRY(fsdevCreateFile(path, FileSize, 0));
R_TRY(fs::CreateFile(path, FileSize));
/* Open the file. */
FILE *fp = fopen(path, "r+b");
R_UNLESS(fp != nullptr, fsdevGetLastResult());
ON_SCOPE_EXIT { fclose(fp); };
fs::FileHandle file;
R_TRY(fs::OpenFile(std::addressof(file), path, fs::OpenMode_Write));
ON_SCOPE_EXIT { fs::CloseFile(file); };
/* Write new header with zero entries to the file. */
LruHeader new_header = { .entry_count = 0, };
R_UNLESS(fwrite(&new_header, sizeof(new_header), 1, fp) == 1, fsdevGetLastResult());
R_TRY(fs::WriteFile(file, 0, std::addressof(new_header), sizeof(new_header), fs::WriteOption::Flush));
return ResultSuccess();
}
@@ -80,36 +81,33 @@ namespace ams::kvdb {
std::memset(this->keys, 0, BufferSize);
/* Open file. */
FILE *fp = fopen(this->file_path, "rb");
R_UNLESS(fp != nullptr, fsdevGetLastResult());
ON_SCOPE_EXIT { fclose(fp); };
fs::FileHandle file;
R_TRY(fs::OpenFile(std::addressof(file), this->file_path, fs::OpenMode_Read));
ON_SCOPE_EXIT { fs::CloseFile(file); };
/* Read header. */
R_UNLESS(fread(&this->header, sizeof(this->header), 1, fp) == 1, fsdevGetLastResult());
R_TRY(fs::ReadFile(file, 0, std::addressof(this->header), sizeof(this->header)));
/* Read entries. */
const size_t count = this->GetCount();
if (count > 0) {
R_UNLESS(fread(this->keys, std::min(BufferSize, sizeof(Key) * count), 1, fp) == 1, fsdevGetLastResult());
}
R_TRY(fs::ReadFile(file, sizeof(this->header), this->keys, BufferSize));
return ResultSuccess();
}
Result Save() {
/* Open file. */
FILE *fp = fopen(this->file_path, "r+b");
R_UNLESS(fp != nullptr, fsdevGetLastResult());
ON_SCOPE_EXIT { fclose(fp); };
fs::FileHandle file;
R_TRY(fs::OpenFile(std::addressof(file), this->file_path, fs::OpenMode_Read));
ON_SCOPE_EXIT { fs::CloseFile(file); };
/* Write header. */
R_UNLESS(fwrite(&this->header, sizeof(this->header), 1, fp) == 1, fsdevGetLastResult());
R_TRY(fs::WriteFile(file, 0, std::addressof(this->header), sizeof(this->header), fs::WriteOption::None));
/* Write entries. */
R_UNLESS(fwrite(this->keys, BufferSize, 1, fp) == 1, fsdevGetLastResult());
R_TRY(fs::WriteFile(file, sizeof(this->header), this->keys, BufferSize, fs::WriteOption::None));
/* Flush. */
fflush(fp);
R_TRY(fs::FlushFile(file));
return ResultSuccess();
}
@@ -209,44 +207,37 @@ namespace ams::kvdb {
return Path::MakeFormat("%s/%s", dir, "kvs");
}
static Result Exists(bool *out, const char *path, bool is_dir) {
static Result Exists(bool *out, const char *path, fs::DirectoryEntryType type) {
/* Set out to false initially. */
*out = false;
/* Check that the path exists, and that our entry type is correct. */
{
struct stat st;
/* Try to get the entry type. */
fs::DirectoryEntryType entry_type;
R_TRY_CATCH(fs::GetEntryType(std::addressof(entry_type), path)) {
/* If the path doesn't exist, nothing has gone wrong. */
R_CONVERT(fs::ResultPathNotFound, ResultSuccess());
} R_END_TRY_CATCH;
if (stat(path, &st) != 0) {
R_TRY_CATCH(fsdevGetLastResult()) {
/* If the path doesn't exist, nothing has gone wrong. */
R_CONVERT(fs::ResultPathNotFound, ResultSuccess());
} R_END_TRY_CATCH;
}
if (is_dir) {
R_UNLESS((S_ISDIR(st.st_mode)), ResultInvalidFilesystemState());
} else {
R_UNLESS((S_ISREG(st.st_mode)), ResultInvalidFilesystemState());
}
}
/* Check that the entry type is correct. */
R_UNLESS(entry_type == type, ResultInvalidFilesystemState());
/* The entry exists and is the correct type. */
*out = true;
return ResultSuccess();
}
static Result DirectoryExists(bool *out, const char *path) {
return Exists(out, path, true);
return Exists(out, path, fs::DirectoryEntryType_Directory);
}
static Result FileExists(bool *out, const char *path) {
return Exists(out, path, false);
return Exists(out, path, fs::DirectoryEntryType_File);
}
public:
static Result CreateNewCache(const char *dir) {
/* Make a new key value store filesystem, and a new lru_list.dat. */
R_TRY(LeastRecentlyUsedList::CreateNewList(GetLeastRecentlyUsedListPath(dir)));
R_UNLESS(mkdir(GetFileKeyValueStorePath(dir), 0) == 0, fsdevGetLastResult());
R_TRY(fs::CreateDirectory(dir));
return ResultSuccess();
}

4
libraries/libstratosphere/include/stratosphere/kvdb/kvdb_file_key_value_store.hpp
View File

@@ -14,8 +14,8 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#pragma once
#include "../os.hpp"
#include "kvdb_bounded_string.hpp"
#include <stratosphere/os.hpp>
#include <stratosphere/kvdb/kvdb_bounded_string.hpp>
namespace ams::kvdb {

77
libraries/libstratosphere/include/stratosphere/kvdb/kvdb_memory_key_value_store.hpp
View File

@@ -15,10 +15,11 @@
*/
#pragma once
#include <sys/stat.h>
#include "kvdb_auto_buffer.hpp"
#include "kvdb_archive.hpp"
#include "kvdb_bounded_string.hpp"
#include <stratosphere/fs/fs_filesystem.hpp>
#include <stratosphere/fs/fs_file.hpp>
#include <stratosphere/kvdb/kvdb_auto_buffer.hpp>
#include <stratosphere/kvdb/kvdb_archive.hpp>
#include <stratosphere/kvdb/kvdb_bounded_string.hpp>
namespace ams::kvdb {
@@ -262,11 +263,9 @@ namespace ams::kvdb {
Result Initialize(const char *dir, size_t capacity) {
/* Ensure that the passed path is a directory. */
{
struct stat st;
R_UNLESS(stat(dir, &st) == 0, fs::ResultPathNotFound());
R_UNLESS((S_ISDIR(st.st_mode)), fs::ResultPathNotFound());
}
fs::DirectoryEntryType entry_type;
R_TRY(fs::GetEntryType(std::addressof(entry_type), dir));
R_UNLESS(entry_type == fs::DirectoryEntryType_Directory, fs::ResultPathNotFound());
/* Set paths. */
this->path.SetFormat("%s%s", dir, "/imkvdb.arc");
@@ -337,7 +336,7 @@ namespace ams::kvdb {
return ResultSuccess();
}
Result Save() {
Result Save(bool destructive = false) {
/* Create a buffer to hold the archive. */
AutoBuffer buffer;
R_TRY(buffer.Initialize(this->GetArchiveSize()));
@@ -353,7 +352,7 @@ namespace ams::kvdb {
}
/* Save the buffer to disk. */
return this->Commit(buffer);
return this->Commit(buffer, destructive);
}
Result Set(const Key &key, const void *value, size_t value_size) {
@@ -468,27 +467,38 @@ namespace ams::kvdb {
return this->index.find(key);
}
private:
Result Commit(const AutoBuffer &buffer) {
/* Try to delete temporary archive, but allow deletion failure (it may not exist). */
std::remove(this->temp_path.Get());
Result SaveArchiveToFile(const char *path, const void *buf, size_t size) {
/* Try to delete the archive, but allow deletion failure. */
fs::DeleteFile(path);
/* Create new temporary archive. */
R_TRY(fsdevCreateFile(this->temp_path.Get(), buffer.GetSize(), 0));
/* Create new archive. */
R_TRY(fs::CreateFile(path, size));
/* Write data to the temporary archive. */
/* Write data to the archive. */
{
FILE *f = fopen(this->temp_path, "r+b");
R_UNLESS(f != nullptr, fsdevGetLastResult());
ON_SCOPE_EXIT { fclose(f); };
R_UNLESS(fwrite(buffer.Get(), buffer.GetSize(), 1, f) == 1, fsdevGetLastResult());
fs::FileHandle file;
R_TRY(fs::OpenFile(std::addressof(file), path, fs::OpenMode_Write));
ON_SCOPE_EXIT { fs::CloseFile(file); };
R_TRY(fs::WriteFile(file, 0, buf, size, fs::WriteOption::Flush));
}
/* Try to delete the saved archive, but allow deletion failure. */
std::remove(this->path.Get());
return ResultSuccess();
}
/* Rename the path. */
R_UNLESS(std::rename(this->temp_path.Get(), this->path.Get()) == 0, fsdevGetLastResult());
Result Commit(const AutoBuffer &buffer, bool destructive) {
if (destructive) {
/* Delete and save to the real archive. */
R_TRY(SaveArchiveToFile(this->path.Get(), buffer.Get(), buffer.GetSize()));
} else {
/* Delete and save to a temporary archive. */
R_TRY(SaveArchiveToFile(this->temp_path.Get(), buffer.Get(), buffer.GetSize()));
/* Try to delete the saved archive, but allow deletion failure. */
fs::DeleteFile(this->path.Get());
/* Rename the path. */
R_TRY(fs::RenameFile(this->temp_path.Get(), this->path.Get()));
}
return ResultSuccess();
}
@@ -505,18 +515,17 @@ namespace ams::kvdb {
Result ReadArchiveFile(AutoBuffer *dst) const {
/* Open the file. */
FILE *f = fopen(this->path, "rb");
R_UNLESS(f != nullptr, fsdevGetLastResult());
ON_SCOPE_EXIT { fclose(f); };
fs::FileHandle file;
R_TRY(fs::OpenFile(std::addressof(file), path, fs::OpenMode_Read));
ON_SCOPE_EXIT { fs::CloseFile(file); };
/* Get the archive file size. */
fseek(f, 0, SEEK_END);
const size_t archive_size = ftell(f);
fseek(f, 0, SEEK_SET);
s64 archive_size;
R_TRY(fs::GetFileSize(std::addressof(archive_size), file));
/* Make a new buffer, read the file. */
R_TRY(dst->Initialize(archive_size));
R_UNLESS(fread(dst->Get(), archive_size, 1, f) == 1, fsdevGetLastResult());
R_TRY(dst->Initialize(static_cast<size_t>(archive_size)));
R_TRY(fs::ReadFile(file, 0, dst->Get(), dst->GetSize()));
return ResultSuccess();
}