Yesterday i was talking about this in the openMSX irc channel, and also i summited a feature request to openMSX on Sourceforge(http://sourceforge.net/tracker/index.php?func=detail&aid=1031805&group_id=38274&atid=421864)
I post this here because i want to know what the MSX community thinks about it.
I think that all MSX emulators must support the TZX format(http://www.worldofspectrum.org/TZXformat.html), we need to have a format that can hold a perfect copy of a Cassette, for preservation purporses. The TZX is the only format that can hold all information for doing a perfect duplicate(not like WAV and Cas). I think that this is important because me and others have got some original strange cassettes and i want to have a backup of it, because a cassette is not forever, and i dont like cracked backup versions out there and bad disk images created from a cassette version which doesn't work right(http://www.msx.org/forumtopic3297.html).
TZX is used in the Spectrum scene and it's their default format for preservation, it can be used for C64, Amstrad(They are using it renamed to CDT to distinguish from a Spectrum Tape image). Also the spectrum scene has got a site where all images can be found(www.worldofspectrum.org) and there are some tools for doing TZX files.
This is the same purpose as the CAPS project(www.caps-project.org), but they want to preserve Amiga disks using their custom format(called IPF*) and in the future other disk systems.I hope the caps project supports MSX disks or the community invented other method to preserve them(RAW backups are evil).
NOTE: The IPF format used for the Caps project can hold information about copy protection making a perfect backup which works like the original disk.
NOTE2: We can rename the tzx extension to other like Amstrad commity does, like for example .tpx(TaPe msX) or whatever.

My proposal is to ASk the MSX community and MSX Emulators developers what they think about this?

Are there any tools to create IPF? I did not find any.
From what I understand from the CAPS Project page, it also stores other information about a game, like cover and manual scans.

There exists a program on MSX called HDDEMU which creates disk images with a header file that includes information about the copy protection. It handles most copy protections used on MSX, and I'm sure it can be extended to handle the last unsupported ones.

For the rest, there's also the FDI format which can store any disk format, so that includes unformatted or otherwise copy protected disks. This format does have a tool available to make diskimages, although it costs a bit money and runs on MS-DOS.

As for TZX... WAV IS a perfect duplicate, and anyone can create it easily. The benefit of TZX would only be smaller files.

Smaller files like WAV? something like... MP3 or OGG perhaps?

hm.. I wonder if MP3, OGG or any other fourrier-based compression wouldn't influence the actual data

FLAC is a compressed wav-format .. doesn't use lossy compression ..

GuyveR800 wrote:
Are there any tools to create IPF? I did not find any.
From what I understand from the CAPS Project page, it also stores other information about a game, like cover and manual scans.

No, there aren't any tools to create IPF, you must send the floppy to the CAPS team and they do the dump.Why? Because they do professional dumps, they don't only dump your disk using theirs tools, they also check that your disk is valid(no bad sectors) and also check that it's no altered in any way(hasn't been writed), and also check your disk against other disks to verify it(if your disk has a bad sector they can also mix in with other disk with the same sector good but others bad, to obtain a good image). This is a warranty of quality.
Their goal is to create dumps of disks that in the future with a special device can be writed back to a disk, like original disk makers did.
With FDI and HDDEMU, you can obtain 2 disks diferent from the same orginal disk. Also FDI exist for the Amiga and it's not a good format.Read more about CAPS deeply.
...And the dumps can only be maken by Amiga Floppy Disks(a good copu can't be made with a PC disk controller or a MSX FDD), the reasen is very long to explain here, again read more about CAPS they say the why.
Also the IPF format dont stores information about scans,covers, ...they distribute it sometimes in the same Zip file.And you can download IPF files fron their page, you must go to IRC or P2P clients to download them(For legal reasons).

GuyveR800 wrote:
As for TZX... WAV IS a perfect duplicate, and anyone can create it easily. The benefit of TZX would only be smaller files.
BiFi wrote:
Smaller files like WAV? something like... MP3 or OGG perhaps?

WAV isn't a perfect duplicate, it contains the original game wave form + noise(this is inevitable due to analog source), for example back to 80s, what is the best form to duplicate a cassette(without protection), using a tape-to-tape recorder or using a copy software?. Better to copy digital data than analog data, isn't?
Also two people can get different wav from the same Game, but digitally it's not possible.
Comparing a WAV format(Holding analog data) VS digital data is like comparing VHS versus DVD, i can do a copy with a VHS but it's not perfect.

Quote:

WAV isn't a perfect duplicate, it contains the original game wave form + noise(this is inevitable due to analog source), for example back to 80s, what is the best form to duplicate a cassette(without protection), using a tape-to-tape recorder or using a copy software?. Better to copy digital data than analog data, isn't?

I must say, that now you are messing together two different things. To get the data to digital form, you have to convert it from analog to digital. If you use wav-format, you will get the data + noice, but how can you say, that the data without noice is more perfect than pure data?

If you want to get a rid of the noice, then just downsample that for example 16bit WAV to 1bit data. MSX can read only 1bit data, so there is no damage done. The noice is anyway part of original data. I forexample remember, that one of my cassette players did not erase the cassette well enough before writeing to that. There fore I could hear the original music "behind" the MSX program. If I would downsample the WAV to 1bit digital data, what I think you are now suggesting, I would lose the music. Sometimes analog data might be even part of the program. (Look : http://www.msx.org/forumtopic3562.html) In this case MSX is the limit, data is much better, than MSX can read.

Quote:

Also two people can get different wav from the same Game, but digitally it's not possible. Comparing a WAV format(Holding analog data) VS digital data is like comparing VHS versus DVD, i can do a copy with a VHS but it's not perfect.

Two people will definately get different WAV file from same analog cassette, but so what as long as MSX can load it? Once the software has been digitized, you can copy it as many times you like without data being changed. WAV is not analog format.

Oh I forget... If you want to get noice free WAV file, you can first copy the WAV to you HDD, then load it on you MSX emulator and then save it again to another WAV file. This way the data does not go though analog format while saveing and you will always get same kind of wav file as long as you use same MSX emulator.

Quote:

No, there aren't any tools to create IPF, you must send the floppy to the CAPS team and they do the dump.

That's ofcourse out of the question.

Quote:

With FDI and HDDEMU, you can obtain 2 disks diferent from the same orginal disk.

o.O
What the hell are you talking about?!

NYYRIKKI:
No i'm not messing together two different things. You are, let's me explain this:
A cassette has DIGITAL data codec in an analog medium.There is not analog data, it's like a modem, you send digital data in a analog medium so we dont need to convert Analog data to digital data, we only need to interpret the digital data. Why everybody wants to hold the digital data in Wav format(It's bigger and can't store other information). A MSX Wav image is a digital representation of an analog data that represents digital data. I bit stupid. This is like when you download a program for internet with your Cable line(Optical fiber) and instead of saving the file you have a wav file which is the digitalization of Laser waveform which represents the program data.(It's possible because optical fiber is an analog medium).

And for disks GuyveR800:
What the hell are you talking about?!, go to caps-project.org and read about it, or in http://eab.abime.net/ you can have more information. But in disks i am not an expert, and i can tell you that an MSX DISK dump must be done with an Amiga Floppy drive(in caps homepage you can find the answer).

Quote:

But in disks i am not an expert, and i can tell you that an MSX DISK dump must be done with an Amiga Floppy drive(in caps homepage you can find the answer).

Basicly you want to read the track data instead of the sector data. The disk controller I know on MSX (the WD17/27) has a read track command, but the output is not reliable. On the Amiga reading a track is synchornised correclty (that's why an Amiga disk can hold 880Kb instead of 720). If you look for the tools the Amiga emulation scene buiit to dump diskimages (or maybe trackimages is a better word) you can see that people did experiment on pc's. Basicly they exported one extra pin from the diskdrive to the printer port that allows them to read tracks with a correct sync (using custom tooling).I tried it once using the adr (amiga diskreader), but I didn't succeed in it. I didn't try the disk2fdi tool, as discussed in www.msx.org/forumtopic2808p15.html.

Afaik, the tzx format is open for extensions, adding a suitable definition for storing MSX tapes should be possible. Maybe a nice first step is to implement cas images in tzx and work from that.

Quote:

And for disks GuyveR800: What the hell are you talking about?!

No, what the fuck are YOU talking about?!!
Go to http://www.oldskool.org/disk2fdi and stand corrected!

Quote:

Quote: And for disks GuyveR800: What the hell are you talking about?! No, what the fuck are YOU talking about?!! Go to http://www.oldskool.org/disk2fdi and stand corrected!

Well, first of all, i'm an expert on floppy disk, i'll tell what i know about.

Disk2dfi can't read all disk formats, you can dump every MSX disk with it. The problem of PC floppy disks are the FDC, but there are tricks to avoid it at some degree. Disk2fdi use these tricks. The Amiga in other hand hasn't FDC and all in done in Software, you can control even the rotation speed and do crazy custom formats. Amiga disks has protections imposible to do on others disk based systems. You can check for example in the caps web their progress(http://www.caps-project.org/wip-jul-2001.shtml, at bottom page you can check other year and moth to see their progress).
Caps project has dumped all those disk which disk2dfi hasn't, you can read about this for example here. http://eab.abime.net/showthread.php?t=4165.

Quote:

DPainter: I was never able to get DISK2FDI to read non-standard tracks except for the occasional error-protected tracks, which I then fixed using a parameter copier in WinUAE. Otherwise it just hangs up on me on the non-standard disks.

But you can't dump disks with Amiga tools, you only can dump disks with Caps dumping tools, and only if they implemented that copy protection(Pratically the vast majority protections are supported, i think).

It's a waste of time discussing this, because over the time, the Caps will support Msx disk, MSX emulators must only support their library and voila, we have now perfect images as the original ones for ALL disks. No more tricks are need to be done to image a MSX disk.

They are know studing/dumping Atari St and IBM PC disks, and as you can read here http://www.atari-forum.com/ftopic654.html&sid=90ac4e4990dd22a98b64cd0a70cebb8e they are waiting to see Caps project supporting their platform to Kick those Cracked, pirated disk images and have a real backup of their original disks.

Please, follow the links, they have usefull information. And you can find more in the Amiga forum http://eab.abime.net/.

And my last word, Disk preservation like Cassette preservation isn't a problem of MSX comunity, it's a problem of Spectrum, C64, Amiga, Atari ST, PC, Amstrad,...I think that all comunities must share their knowledge and use a common format, TZX for tape images perhaps(As Amstrad and Spectrum comunity does and works well) and CAPS for disks, although CAPS is not an open format(all can change) but it's now the default disk format, over ADF, DFI, and others...

Also i forgot to post a link which tecnically describes somethings

http://www.atari-forum.com/ftopic3454.html

It's very big, and i think you can't read all(i can't).

Hi, just come to put a couple of things straight, please take no offense.

zabado78 said:
>No, there aren't any tools to create IPF, you must send the floppy to the CAPS team and they do the dump

This is not correct. All you need to have is a suitably equiped Amiga, and we send you the software and you dump it

It is *really* not a good idea to send magnetic media through the post if you do not have to.

>The Amiga in other hand hasn't FDC and all in done in Software, you can control even the rotation speed and do crazy custom formats.

Unfortunately, you can't control the rotation speed of the Amiga drive. However, it is tolerant to changes of about 15% either way, so the *duplication machines* made disks with data recorded "at different speeds" (quoted, because that is not quite right, the true answer is that the bitcells written are of different widths than normal) and the Amiga can read (but not write) data like that.

>But you can't dump disks with Amiga tools, you only can dump disks with Caps dumping tools, and only if they implemented that copy protection(Pratically the vast majority protections are supported, i think).

Not correct. The dumping software has been able to dump every disk we have tried so far. We need to support formats so we can see if the data is correct. And this is done after the game is dumped in our analysation software. Doing dumps is easy, describing their formats/protecton can be very difficult. However, for the Amiga, as you say, we support pretty much everything now.

Thanks,

Kieron
The CAPS Team
http://www.caps-project.org

I've got one silly question tho... What happens after the dumps are made and analyzed? Do they just sit there on a shelf? Who's in charge of making sure these titles aren't "lost forever" and in which way are they stored?