Random Stuff

MetaTrader History ToolKit
My ATC2012 EA 
Heinz Engine
MetaResources 
Honda Steed Exhaust Mod
ResMagik
Frypan-Lamp Biquad Antenna
HTPC 001
Aerospace AL CPU Cooler 
PHP Pager
de-db 
Developer's Notes

MetaResources

(Technology Preview Page)

MetaResource(s) is a binary file format for "resource files". Its purpose is to easy the localization of programs, development of applications in multiple languages and deployment of logical data.

The most eye-catching and unique features are its runtime Object-Oriented approach and its dynamic polymorphism characteristics, like: Resource Aliasing, Resource Inheritance, Resource Overriding and Resource Overloading. These features are reviewed later on this page.

The project has been created from scratch and has been an integral part of Heinz Engine since its creation. It is now under its next development iteration (v1.1 mature alpha) and I am now thinking of making a stand alone public version to be used as a universal resource file format, where programmers can use the official implementation(s) or they can create their own by following a defined set of specifications and conventions.

The official implementation defines the following file extensions:

  • mrf, mr: MetaResource file. This is the final output resource file.
  • mrs, ms: MetaResource script. This is the script file used to compile a resource file.

What are resource files and why should you use them?

The answers are quoted from ResMagik's documentation page here.

What's a resource file?

Most applications use text, images, sounds, videos and many other external files or data. These objects are also known as resources. These resources are used for multiple purposes depending on the type they are. As you might already noticed, many programs you have downloaded use images but, where are these images stored if the programs are only single executable files and sometimes they come with a few satellite assemblies not even related to an image file? Well, the only answer is that these programs are using resource files. Resource files then are files just like any other file but they act like "packages" to store other files, programs are able to read these packages and retrieve data. There're many custom engines for data retrieval but the main concept and popularity of resource files born with the old Win16 resource. Probably, you're now asking yourself: Why should i use a resource file if there're many cool packaging file formats out there? Because compression formats such as RAR, ZIP, ACE, TAR, etc are used only to package or compress files while resource files store many other data types (including files) such us: Strings, dialogs, menus, accelerators and many other serializable objects. Resource files can be used in web sites (dynamic pages). Resource files differ from other file types in the way they can be used: Resource files may be used directly by an application or (one of the most important feature) they can be embedded inside executables or libraries (the executable reads itself to retrieve data).

"...A resource is any nonexecutable data that is logically deployed with an application..."
.NET Framework Developer's Guide.

Why to use a resource file?

  • Because output text (legible by the user) inside your source files, is "mixed" with the source code. Finding a specific string inside a huge source file can be very tedious and time consuming. Suppose you want to do a spell check, you'd have to find every string in the source and then correct them. If you use resource files, all your strings are stored in a single place (table), making corrections and translation very much easier and faster.
  • Reduce file size by storing repeated strings and objects only once.
  • Because files get packed and the distribution of your software is more efficient. Otherwise when you distribute your software, you have to make sure that every file is available and in the right path. Also if you don't use resource files you'll probably have to distribute hundred of files along with your main executable and libraries, filling your hardrive or file system with files and entries.
  • Because resource files help developers to create applications with multilanguage support. Resources are localized with a specific culture building translated versions of the program.
  • Because resource files secure your data in a very low manner by embedding it with satellite libraries and executables. And protect web content from direct download.
  • Because file access is much faster:

    NOTE: This was written in 2007. Today we have SSDs that mitigate seek and read latency, but there is still the overhead of polling the filesystem for every single file plus triggering the OS file access chained events.

MetaResource concepts

The MetaResource API understands the concept of a ResourceManager, Language, Folder, Resource, MetaResource and MetaData. A ResourceManager is the core object, it represents a 'physical' resource file and it is used to manage operations with all the other objects. A ResourceManager contains one or more Languages, which are exactly what they mean. A Language is the root localized object that can contain resources directly and folders that in turn can contain resources and sub-folders. Folders are flexible sub-containers that allow to create tree-like structures, making design time representation possible by using many tools like: XML, Windows registry, a file system and every software technology capable of working with trees. A Resource represents logical data in its complete state (resource data + meta data). And finally...
What is a MetaResource? A MetaResource is an arbitrary data object handler that in combination with other information becomes simply a "resource". A resource, internally has a "MetaData" associated with it that gets embedded into the resource file, then this MetaData can be read back by using the corresponding MetaResource or even be "metamorphosed" by overriding the MetaResource information. A MetaResource also describes how resource data is written and read to/from the resource file and perform other operations on the internal MetaData.

MetaResource general features

  • Latest version has been developed exclusively for C++11. I'm looking for a way to make a C client for interop with other languages and platforms.
  • Runs theoretically in every computer system, platform and device capable of setting up a C++11 runtime: Windows, FreeBSD, Linux, MacOS, Android, PlayStation, Xbox, Nintendo Switch, etc...
  • Dynamic binary file structure optimized for the best ballance between smallest file size and read performance. Comes from a game engine background where performance and optimizations are priorities. It is now successfully being used for general applications.
  • Dual official implementations: There is the dynamic structure one that helps to reduce file size (great for embedded systems) and there is the fixed size structure one that takes more space but helps a little bit with read performance (usually not noticeable at all).
  • Flexible and extensible specification. Custom resource types can be embedded by implementing the MetaResource interfaces. What type of data can be embedded is limitless.
  • It comes with a complete suite like the resource I/O API, a command line resource compiler and a GUI resource editor.
  • In version 1.0, a single resource manager class was used to write and read resource files, making it very easy to create frontends for the API. But in version 1.1 the library has been split into two namespaces: One for writing and the other one for reading resource files, this results in the best approach for a more lightweight (uses less RAM) and even further optimized (for read speeds) implementation, which was the priority in this new version.
  • The write namespace provides all the static polymorphism functionality like static analysis, type checking and alias resolution (for optimization). While the read interface provides all the runtime and dynamic polymorphism functionality like inheritance, overrides, aliasing, MetaResource and MetaData handling.
  • Per resource compression and encription, allowing private critical resources to be obfuscated while leaving fast read critical resources untouched.
  • Per folder aliasing. This allows for selective inheritance and overriding of folders and resources. Think of it like a "sub-inheritance": Like a C++ class can derive from one or more other classes inheriting their members, now the members can in addition inherit and override the members of other sub-members.
  • It comes with built-in MetaResource types registered for the most commonly used data types. This can be extended arbitrarily by the programmer, making the file format an all-around file format: It can be used to create for example video, audio, maps, model and everything file format! And with localized data bonus!.
  • The built-in MetaResource "STRING" and "STD_STRING" store strings in UTF-8 encoded streams, performing sequence validation at compile time. But any other string encoding can be embedded.
  • Implicit optimization of duplicated data.
  • Resources are usually referenced by the programmer only once and then they are used for the rest of the program's lifetime. Nonetheless, the official implementation comes with a built-in cache system that caches only referenced elements: It loads what you use, not the entire file into system memory. This helps a lot with the seek times for elements referenced more than once.
  • Inherited elements can be optionally cached and even be pre-cached. Useful for improving seek performance in complex aliased structures.
  • Resource files can be loaded not only from physical files but from any arbitrary source like: Memory streams, network streams, Win32 resource files (yep, that's inception right there, a MetaResource file embedded into a Windows resource file, and quite useful might I add) and any custom readable byte stream.

MetaResource key features and concepts

The MetaResource file format provides a smart and fast way to design multi-language programs with the introduction of the following dynamic polymorphism concepts:

Resource Aliasing

Resources have a property called "Alias" that can be set to point to an equivalent resource located at the same path, in another language. When you set a resource's alias you are "linking" that resource to its alias, making it to reference its alias's data instead of its own.

Consider the following example: You have created an application in German language. Time after that you decided to make an english translation of the program with the ability to switch between the two languages. So, with MetaResources what you do is add every single entry in the MetaResource file from German to its English equivalent. You get a structure like this one:

When the program is set to run in English then it will fetch all the corresponding assets from the English language table. You expect the application to look something like this:

That is standard resource file behaviour. But you could put Resource Aliasing into play: You could understand that the German and English language tables each one contain their own resources in their respective culture, very different one from another. But there are elements in common between the two: For example the program name is unique and is the same for both languages. There is also the image, that one doesn't change no matter the language. So, what you do in MetaResources is to set the alias of the resources "Program_Name" and "Screen1_Image_1" in the English language to point to their German language equivalent. This will explicitly optimize storage space and memory usage by referencing only a single data instance. This is a simple diagram of how resources reference each other:

Running the program would yield the same looking application but with the previously mentioned benefits/optimizations. All of the resources would be loaded from the English language table except for the program name and image, those would be loaded from the German language table (where the actual data is located).

Even more complex cases of Resource Aliasing can be created (if you need to, just because). Suppose you create a MetaResource file with 4 languages for an application. The program name would be the same for all languages, so the most obvious thing to do is to set the alias of 3 languages to point to the resource of a single base one. That would be correct and efficient, but it is also possible to do a "chained aliasing" by setting the resource alias of the last language to point to the resource of the previous language, then set the resource alias of that previous language to point to the resource of its previous language...and like that causing a chain reaction until the alias resolves to the base resource where the data is actually stored. This diagram shows how this chaining would look like:

In the example above, if the application was run in Finnish language then it would try to get the program's name. That resource is aliased to the Italian language and that Italian resource is aliased to the English language and that English resource is aliased to the German language where it is actually stored, finally resolving and fetching the data. This chain reaction seems somewhat inefficient but it is not, at compile time the API is able to optimize away this chaining and resolve linking statically.

So, Resource Aliasing means "Resource Linking" (actually the term is still a candidate for the final release, together with the "Link" property instead of "Alias").
Aliasing or linking is the core step to enable certain dynamic polymorphism features when applied to elements other than resources. We will review them next.

Resource Inheritance

Resources are not the only elements that support the concept of aliasing. Languages and folders also support aliasing. When the "Alias" property of a language or folder is set, they are instructed to "behave like their equivalent alias language/folder", this mean that a language or folder can have its own child sub-folders and resources but also must inherit the ones from its alias equivalent.

When is inheritance used? Consider the following example: You are developing an application in multiple languages, each one with its own localized resources. Among these languages there is "en-US" (English, United States) and you want to add for personal reasons the language "en-GB" (English, Great Britain). But it just happens that at the current program' state, all resources for both of these English languages are equal, hence it would not be practical to duplicate all the resources from en-US into en-GB. The solution is to set the "Alias" property of en-GB to point to en-US, this way it will inherit all of the children from en-US, callable from en-GB, while optimizing storage space and memory usage. Such a structure would simply look like this:

More flexible or complex structures can be created by aliasing folders, this enables selective inheritance by importing only members from aliased sub-folders. Such a hybrid structure could look like this:

Chained aliasing is also supported on languages and folders, recursively inheriting the corresponding members and further extending the complexity of structures that can be created.

Resource Overriding

Aliased languages and folders inherit members from their respective aliases and they are also allowed to have their own. As a rule of hierarchy and inheritance, if a direct member has the same name of an inherited member then the first overrides the second. This is quite useful for languages that have many in common but want their differences to prevail. Lets bring this to an example:

You are developing an application in English language. But you want to cover a wide spectrum of English cultures so you create a MetaResource file with support for en-US (English, United States) and en-GB (English, Great Britain). Language en-GB has been aliased to en-US and inherits all of en-US members because both languages have all their resources in common, except for this one string that in British English is written differently than in American English: "Choose a colour" (en-GB) vs "Choose a color" (en-US). In such case, en-GB will provide its own string and override the inherited one. Such structure is easy to implement and would look something like this:

In this example, if the program is run with the en-GB language then it will load all the resources from en-US except for the overriden string that en-GB provided. Such application, if run in both languages, would look something like this:

Here is another, more complex example: Inherited members can also be overriden with other inherited members:

In the example above, the language de-DE is aliased and points to the language de-AT, inheriting all of its members. Then de-DE inherits the folder "Folder1" with two resources in it. One would think that those resources are obtained from de-AT but that is not correct because de-DE overrides its inherited "Folder1" from de-AT by providing its own "Folder1" that is aliased and points to "Folder1" located at de-CH. So, the inherited resources from de-AT get overriden by the inherited resources from de-CH because they come from a higher level alias. If de-DE would have provided only its own "Folder1" but without it being aliased then it would have inherited the resources from de-AT because there would have been no other resources (inherited or not) that would override the inherited ones.

When there is chained aliasing present on a language or folder, members are recursively inherited and the same overriding rules are perfectly applied for each and every single element, ascending or descending, in all levels and dimensions.

Resource Overloading

This is an experimental feature that will for sure be excluded from final release. It allows languages and folders to have multiple resources of different types but with the same name. This is pretty much equal to function overloading in C++ where multiple versions of a function with the same name but different parameters can coexist. This brings little benefit and extra overhead because it requires the resource type to be specified in every function for disambiguation.

Using them all together

All of these features are compatible between each other and can be used all at the same time, multiple times, at different levels and dimensions. The degree of complexity that can be achieved by a MetaResource structure is beyond imagination.

The most common workflow for creating a multi-language structure would be:

  1. Create a "NEUTRAL" language. This would be a master language that is the maximum common denominator and contains all the resources that all languages have in common.
  2. Create base culture languages and alias them to point to the neutral language to inherit the base resources. Plus add the localized resources.
  3. Create derived languages and alias them to point to their corresponding base culture language to inherit all of the neutral resources plus the localized ones. Additionally apply resource overrides specific to the derived culture.

History

The past

The first MetaResource file format experiments started in 2007 together with my first game engine studies, but it was in 2010 when the MetaResource API got its implementation matured and widely used for private custom programs. It was indeed created as a deployment solution for the engine, allowing it to pack all the resources and assets (text, sound files, texture files, model files, level data, custom binary data, etc...) into a single resource file. It was mainly used for GUI together with the engine.

How much the overall project has evolved is amazing. It is more flexible, simpler to use, powerful and faster than ever! And still using it for whatever program I do (even for creating non game nor multimedia apps).

The future

Today I'm thinking of a way to make it a public, general purpose resource file format "and specification" but the amount of time required is outstanding. The integration with the game engine is something that can be overcomed but still requires re-engineering. And it will require a native C implementation (or at least a client) that is a lot of work. Plus the overall time and money to setup the project, website+domain, the documentation, the support, while still trying to eat and survive......is really expensive. The idea of MetaResources being widely adopted worldwide as the universal resource file format, programmers getting to know and taking advantage of its unique features is just hearthtaking for me. But I might have to come to realization that it may not come sustainable at all and this is just an utopic idea...in the mean time I leave this page as a "Technology Preview Page".


Thank you for your time and interest to read this article.
This is the kind of things I invent in my free time. I just follow my intuition: I can hear/feel the god(s) in my mind/heart telling me to do this stuff and how to do it.