Organizing data in ROM

By Arjan

Paladin (767)

Arjan's picture

04-04-2022, 23:36

A few months ago I published makelib, a tool that combines a bunch of files into one big so-called library file. Such a file can be easily used in disk-based games and has some nice advantages, like overcoming the 112 files limit and reducing the amount of wasted disk space.

Recently I thought it would be cool to be able to use such a LIB file in a (mega)ROM, so I created a simple program as a proof of concept.

After that I took a bigger step, turning SD Yab Yum into a megarom. The disk version of SD Yab Yum preloads all files in memory, and when the game requires a specific file, it calls a load function with just a file number and a destination address to load that file into memory (decompressing the preloaded file to the destination address). Earlier versions of the game actually loaded the files from disk every time using a similar function call.

To make it work from a megarom, I wrote a new loader (to initalize stuff like rom/ram slot info) and a new function for loading files (basically combining the functions from the example above and some code I already had in the game). The interrupt routine also required a small change to make sure the correct rom/ram is selected/restored, but other than those changes, there were no additional changes needed to make the game work.

So... to get back to organizing data in ROM: a LIB file can be easily used for megaroms but the metadata in it is not really optimal:
- The first two bytes indicate the number of files in the LIB, useful for loading the rest of the metadata from disk but not needed in a ROM.
- For each file in the LIB, there's a 6 byte 'directory' entry: 3 bytes for the location (offset) of the file in the LIB, and 3 bytes for the file size.

The offset is fine for a disk based program, but in a megarom it requires some extra calculations to determine which segment the data is in and the position of the data in that segment. Ideally, this should be precomputed and stored in the LIB to be able to switch files into memory even faster.

The file size is 3 bytes (so you can put an OPL4 samplekit in a LIB, if you'd like...) but it could also be 2 bytes or maybe not needed at all. The ROM version of SD Yab Yum doesn't need the file size anyway since all data is compressed and after decompression the game knows the original file size.

Sometime soon I'll make a version of makelib with some extra configuration options (selectable segment size 8KB or 16KB, which segment to use first, options to omit some metadata).

I don't know how everybody else wants to access their data in ROM... Maybe a symbol file with for each file labels for segment number and address?

Any other ideas?

Login or register to post comments

By ro

Scribe (4708)

ro's picture

05-04-2022, 20:18

Hai, are you aware that Fuzzy Logic has such .lib container file since 1993 already?
Just check out the Muzax series for example Smile

Perhaps, to terminate confusion, you'd call it differently. Not a library .lib file

edit: while reading a bit further... are you sure you aren't using F-kernel (Midas) from the fuzz? what you describe is exactly how Midas works Smile (or atleast, the disk, ram and expand functions)

By Arjan

Paladin (767)

Arjan's picture

06-04-2022, 09:41

Yeah I know, F-Kernel is used in stuff like Muzax, Sunrise Magazine, Umax games... But I'm not using it Tongue I wrote my own routines after reading some articles on some Sunrise Magazine edition in the previous century Tongue For SD Yab Yum I rewrote the whole thing, created a PC tool to speed up development. Of course I also wrote some memory management routines, but I wouldn't really call it a kernel, I guess F-kernel goes way beyond what I developed (I haven't seen F-Kernel, tries googling it but thefuzz.nl is offline).

By ro

Scribe (4708)

ro's picture

06-04-2022, 14:46

Yeah, fK was indeed used in several thingies. It is intended as an OS. It has base memory management and file management and what not. As a matter of fact, I've been working on the next version of the kernel since Januari 2021.. only like 17 years after the last release. haha

I'ts funny to read you do exactly the same things. Still wouldn't mind if you change the "library" .lib context, tho.

oh, and indeed a good reminder to have thefuzz.nl fixed (lazy person here)