Support .n64

Discuss ROM-compatibility related issues here.
Post Reply
User avatar
tony971
Posts: 15
Joined: Sun Feb 01, 2015 1:02 pm

Support .n64

Post by tony971 » Sun Sep 20, 2015 10:07 am

It's regarded as the official format for ROM dumps since it's the type used in the official SDK. .z64 is viewed as a byte-swapped format.No-Intro uses .n64 and therefore RetroArch will only recognize .n64 formats. It would be a real shame to have to have two copies of each game.
Last edited by tony971 on Sun Sep 20, 2015 12:04 pm, edited 1 time in total.

User avatar
iwasaperson
Posts: 49
Joined: Tue Apr 22, 2014 12:50 am

Re: Support .n64

Post by iwasaperson » Sun Sep 20, 2015 11:14 am

tony971 wrote:It's regarded as the official format for ROM dumps since it's the type used in the official SDK. .z64 is viewed as a byte-swapped format. They're both big-endian but you seem to have the minority opinion for which one is correct. No-Intro uses .n64 and therefore RetroArch will only recognize .n64 formats. It would be a real shame to have to have two copies of each game.
I'm pretty sure Nintendo uses z64 for VC. GoodN64 is all z64, and RetroArch supports z64. Even in other N64 emulators, they have to byte-swap N64 ROMs on load.

User avatar
tony971
Posts: 15
Joined: Sun Feb 01, 2015 1:02 pm

Re: Support .n64

Post by tony971 » Sun Sep 20, 2015 12:03 pm

iwasaperson wrote:I'm pretty sure Nintendo uses z64 for VC. GoodN64 is all z64, and RetroArch supports z64. Even in other N64 emulators, they have to byte-swap N64 ROMs on load.
RetroArch bases its game recognition off of checksums. As soon as you byte-swap out of .n64, these checksums change and RA no longer detects the game. So without non-official tools, you break compatibility with RetroArch by not supporting .n64. Also, No-Intro is generally regarded as better than GoodN64. Lastly, .z64 uses the original big endian format, but .n64 is what Nintendo used in their SDK.

https://en.wikipedia.org/wiki/List_of_N ... le_formats

https://github.com/libretro/RetroArch/issues/1963
Last edited by tony971 on Sun Sep 20, 2015 2:48 pm, edited 4 times in total.

ShadowFX
Posts: 86
Joined: Sat Oct 05, 2013 2:08 am
Location: The Netherlands

Re: Support .n64

Post by ShadowFX » Sun Sep 20, 2015 2:17 pm

After reading up on the source material you provided, I would say that supporting both .z64 and .N64 is the way to go with the ROM data stored in the N64's native big endian byte-order. What I don't yet understand is, how you can differentiate between .n64 and .N64?

Eventually supporting both .z64 and .N64 would make sense as those two are the formats compatible with the design philosophy of CEN64 (see http://forums.cen64.com/viewtopic.php?f=9&t=177).
"Change is inevitable; progress is optional"

OS: Windows 10 Pro x64
Specs: Intel Core i7-7700K @ 4.2GHz, 16GB DDR4-RAM, NVIDIA GeForce GTX 1080 Ti
Main build: AVX (official)

User avatar
tony971
Posts: 15
Joined: Sun Feb 01, 2015 1:02 pm

Re: Support .n64

Post by tony971 » Sun Sep 20, 2015 2:57 pm

I'm not sure how to differentiate between .n64 and .N64. The file extensions aren't case-sensitive. I think you'd actually have to check the contents of the ROM.

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Support .n64

Post by beannaich » Sun Sep 20, 2015 5:16 pm

ShadowFX wrote:What I don't yet understand is, how you can differentiate between .n64 and .N64?
You can examine the first 4 bytes of the ROM image, since they should™ be consistent across all images. Most of the front ends I've seen for CEN64 handle the byte-swapping for you, and I remember MarathonMan saying that it's simpler to only accept Z64.

User avatar
tony971
Posts: 15
Joined: Sun Feb 01, 2015 1:02 pm

Re: Support .n64

Post by tony971 » Sun Sep 20, 2015 5:25 pm

beannaich wrote:You can examine the first 4 bytes of the ROM image, since they should™ be consistent across all images. Most of the front ends I've seen for CEN64 handle the byte-swapping for you, and I remember MarathonMan saying that it's simpler to only accept Z64.
Well let's find out. @Presence how difficult would it be to byte-swap big endian .n64 files on the fly? (As in not permanently save them as .z64 files)

Edit: Apparently tagging doesn't work on this forum.

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Support .n64

Post by beannaich » Sun Sep 20, 2015 5:38 pm

tony971 wrote:how difficult would it be to byte-swap big endian .n64 files on the fly? (As in not permanently save them as .z64 files)
I did it when Breadwinka had his own front end written in C#. It added no noticeable overhead to loading a ROM, and was easily implemented. The only part that sucked was you still had to save the file so you can pass it to CEN64 via command line arguments.

I opted for an auto-named temp file, NOT a randomly generated file name unless you want hundreds of MiB of temp files. "/tmp/cen64.tmp.z64" or similar.

User avatar
tony971
Posts: 15
Joined: Sun Feb 01, 2015 1:02 pm

Re: Support .n64

Post by tony971 » Sun Sep 20, 2015 6:11 pm

beannaich wrote:I did it when Breadwinka had his own front end written in C#. It added no noticeable overhead to loading a ROM, and was easily implemented. The only part that sucked was you still had to save the file so you can pass it to CEN64 via command line arguments.

I opted for an auto-named temp file, NOT a randomly generated file name unless you want hundreds of MiB of temp files. "/tmp/cen64.tmp.z64" or similar.
Well if you need temp files anyway, then the better option would probably be to have Cen64 support big endian .n64 files natively. Because with No-Intro/Retroarch exclusively supporting it, that file type isn't going away. Cen64 might actually be viewed as using the wrong file type for exclusively supporting .z64.

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Support .n64

Post by beannaich » Sun Sep 20, 2015 9:34 pm

tony971 wrote:Well if you need temp files anyway, then the better option would probably be to have Cen64 support big endian .n64 files natively. Because with No-Intro/Retroarch exclusively supporting it, that file type isn't going away. Cen64 might actually be viewed as using the wrong file type for exclusively supporting .z64.
I didn't say temp files, I said temp file (singular) :P

Regardless of the amount of file formats supported "natively", the emulator will internally use only one. This is true of all emulators, regardless of how many file formats they support (unless the emulator is terribly designed). When CEN64 was designed, it was designed to use a format that would most easily integrate with the host processor (typically, little endian). Since that is the case, it makes sense for the CEN64 backend to demand the images be in the same format. While relying on a certain endian-ness can absolutely cause issues for portability, CEN64 is just not mature enough for this to be an issue. There are much bigger fish to fry currently, like getting games to run at playable speeds.

But, in all honesty, it would take a few lines of code to support .n64 format in the core, I just don't see it happening until the emulator is more mature and usable. Maybe submit a pull request, or create a feature request on github?

User avatar
tony971
Posts: 15
Joined: Sun Feb 01, 2015 1:02 pm

Re: Support .n64

Post by tony971 » Mon Sep 21, 2015 1:22 am

Both .z64 and .N64 are big-endian.

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Support .n64

Post by beannaich » Mon Sep 21, 2015 9:53 am

tony971 wrote:Both .z64 and .N64 are big-endian.
I honestly had no idea that there was a difference between .n64 (lowercase) and .N64 (uppercase) formats; what a nightmare N64 emulation is. Either way, my last comment still stands. The emulator chose a format to use internally, and until bigger issues are sorted out (or until someone else adds it), you probably won't see .n64 byte swapping in CEN64.

Or until marathonman randomly adds support for it, to make me look like a tool. :p

Presence
Posts: 51
Joined: Fri Oct 18, 2013 9:27 am

Re: Support .n64

Post by Presence » Mon Sep 21, 2015 11:14 am

It's trivial to add support for it, as beannaich said. Would work just the same as the converter in the frontend, just saving the ROM to a temporary location instead and telling CEN64 to launch it from there.

My position on this (and a lot of other things) was/is to employ a wait and see approach and see what direction MarathonMan takes with the core. There's no point in using my time to code things into CEN64-Qt that will eventually be supported in the core.

User avatar
wareya
Posts: 16
Joined: Tue May 19, 2015 5:44 pm

Re: Support .n64

Post by wareya » Tue Sep 22, 2015 6:45 am

Little endian nintendo 64 ROMs were a mistake, and it's yet another mistake for no-intro to use them.

User avatar
Breadwinka
Posts: 54
Joined: Fri Oct 04, 2013 11:35 pm

Re: Support .n64

Post by Breadwinka » Tue Sep 22, 2015 12:31 pm

Presence wrote:It's trivial to add support for it, as beannaich said. Would work just the same as the converter in the frontend, just saving the ROM to a temporary location instead and telling CEN64 to launch it from there.

My position on this (and a lot of other things) was/is to employ a wait and see approach and see what direction MarathonMan takes with the core. There's no point in using my time to code things into CEN64-Qt that will eventually be supported in the core.
Exactly thats what I would do to. Specially since were getting another rewrite. The code for it was not very long at all I wish I still had the source to show everyone how simple it is but lost it when my hard drive failed.

Presence
Posts: 51
Joined: Fri Oct 18, 2013 9:27 am

Re: Support .n64

Post by Presence » Tue Sep 22, 2015 10:52 pm

The code for the converter is here (uses the Qt classes QFile and QByteArray):
https://github.com/dh4/cen64-qt/blob/ma ... er.cpp#L54

It reads and writes 1KB at a time since a couple quick tests on my machine found that to be close to the sweet spot for speed.

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Support .n64

Post by beannaich » Fri Sep 25, 2015 11:33 am

Presence wrote:It reads and writes 1KB at a time since a couple quick tests on my machine found that to be close to the sweet spot for speed.
Sounds about right, I know Microsoft recommends 4KiB for file buffers. But in my personal tests over the years, 1-4KiB seems to be the sweet spot. Regardless, the process seems to happen instantaneously on modern machines. Especially once your FS/HDD caches the file data. :)

User avatar
MarathonMan
Site Admin
Posts: 692
Joined: Fri Oct 04, 2013 4:49 pm

Re: Support .n64

Post by MarathonMan » Fri Sep 25, 2015 9:53 pm

The file extension doesn't matter; CEN64 takes any big-endian formatted ROM image.

Little-endian format will never be supported. My personal feelings are that it's silly to store ROM images in non-native format.

I will outright reject any pull requests that try to implement support for little-endian ROMs.

User avatar
iwasaperson
Posts: 49
Joined: Tue Apr 22, 2014 12:50 am

Re: Support .n64

Post by iwasaperson » Sat Sep 26, 2015 11:41 am

MarathonMan wrote:The file extension doesn't matter; CEN64 takes any big-endian formatted ROM image.

Little-endian format will never be supported. My personal feelings are that it's silly to store ROM images in non-native format.

I will outright reject any pull requests that try to implement support for little-endian ROMs.
I usually use No-Intro dumps, but I was forced to use GoodN64 and delete everything that wasn't a good dump and wasn't US instead.

How do we convince No-Intro to use big-endian dumps?

User avatar
izy
Posts: 25
Joined: Tue Jun 02, 2015 11:34 am

Re: Support .n64

Post by izy » Sun Sep 27, 2015 9:05 am

Once you've put a N64's ROM image into the ram memory, you can swap the words there. It is something easy, fast. Two lines of code. Thats what is required to turn an array from little to big endian, or vice versa.
I saw that CEN64 memory-maps the ROM file being used. Prove me wrong as i say that that won't give any advantage over a full read of the ROM image file into the volatile memory. Actually, if anything, that is a disadvantage... for example all x86 PREFETCH instructions won't work against memory mapped files. This behavior is easily visible using mincore(2) and is the same reason why a PREFETCH won't do anything if the target address is placed in the swap partition, basically a big mmap'ed file (madvise(2) is a sort of very high level replacement for PREFETCH)
The KISS keep it easy interface is the way to go:
malloc|posix_memalign|mmap|VirtualAlloc+...
in order to allocate the destination MEMORY followed by either...
...+fopen+fread INTO MEMORY+fclose+POSSIBLY byteswap=result in MEMORY
...+mmap|MapViewOfFile+POSSIBLY decompress OR plain memcpy INTO MEMORY+munmap|UnmapViewOfFile+POSSIBLY byteswap=result in MEMORY
No temporary files are needed. Following that logic you get the support for compressed files (such as lzma2) and images of any endianness. All will always work and you won't have anymore to care about the way the ROM was made 8-)

User avatar
MarathonMan
Site Admin
Posts: 692
Joined: Fri Oct 04, 2013 4:49 pm

Re: Support .n64

Post by MarathonMan » Sun Sep 27, 2015 2:50 pm

iwasaperson wrote:How do we convince No-Intro to use big-endian dumps?
Simple: Stop using them.
izy wrote:Once you've put a N64's ROM image into the ram memory, you can swap the words there. It is something easy, fast. Two lines of code. Thats what is required to turn an array from little to big endian, or vice versa.
Sure, but by the same token, a two-line C program can be written to byteswap the ROM into its native format once and for all. ;)
izy wrote:I saw that CEN64 memory-maps the ROM file being used. Prove me wrong as i say that that won't give any advantage over a full read of the ROM image file into the volatile memory. Actually, if anything, that is a disadvantage... for example all x86 PREFETCH instructions won't work against memory mapped files. This behavior is easily visible using mincore(2) and is the same reason why a PREFETCH won't do anything if the target address is placed in the swap partition, basically a big mmap'ed file (madvise(2) is a sort of very high level replacement for PREFETCH)
The KISS keep it easy interface is the way to go:
malloc|posix_memalign|mmap|VirtualAlloc+...
in order to allocate the destination MEMORY followed by either...
...+fopen+fread INTO MEMORY+fclose+POSSIBLY byteswap=result in MEMORY
...+mmap|MapViewOfFile+POSSIBLY decompress OR plain memcpy INTO MEMORY+munmap|UnmapViewOfFile+POSSIBLY byteswap=result in MEMORY
No temporary files are needed. Following that logic you get the support for compressed files (such as lzma2) and images of any endianness. All will always work and you won't have anymore to care about the way the ROM was made 8-)
For desktops, it probably doesn't make much of a difference. I chose the solution because it allows the operating system to gracefully page out the ROM image if the user has a particularly small amount of memory (on a smartphone, perhaps, or on my anemic VM on which I test things!), and because this is the very reason why memory-mapped files exist. Either way, the backbone of the PI is only a 16-bit interface IIRC, so performance *really* isn't an issue here.

That being said: it's a bit of a "moral crusade" on my part: the more developers are willing to put up with these non-native the format files, the longer they will exist and the more crud everyone will need to fill their applications with. You might say, "so what, it's just a few swaps", but think about things like the 64drive. Everytime I want to copy a image over UART, I now have to make sure the image is swapped in the correct format before sending it.

Another reason to avoid byteswapping: look at most "detection" algorithms and you'll see all they do is check the first few bytes of the ROM to see if they match 0x8037_1240 (or some variation of those bytes, in the event the image is byteswapped). If you look at the header spec, this is really just the PI DOM1 registers (which need not be those values!) Those values are just what the commercial carts tend to use, so everyone else follows suit. But! You can easily make a cart that boots on the console, yet doesn't use those values. But in doing so, you've now just broken (probably all, if not several) emulators which try to read the first few bytes to determine endianness. Even PJ64 is guilty in this regard: https://github.com/project64/project64/ ... s.cpp#L223

N64's history is filled to the brim with hacks layered on top of hacks and I won't contribute to or tolerance such nonsense.

There's simply no reason for more than one format to exist, and I believe big-endian is the "right" one, and as an emulator author I choose to use my position to advocate for the adoption of one file format.

User avatar
Snowstorm64
Posts: 303
Joined: Sun Oct 20, 2013 8:22 pm

Re: Support .n64

Post by Snowstorm64 » Sat Oct 10, 2015 6:35 pm

MarathonMan wrote: There's simply no reason for more than one format to exist, and I believe big-endian is the "right" one, and as an emulator author I choose to use my position to advocate for the adoption of one file format.
Well said. I would add that although it looks a trivial thing, it is not because they have done a serious disservice to the N64's preservation by uploading and spreading byteordered ROMs that aren't the very same ones that are found in the N64 cartridges, and that those ROMs wouldn't boot on a real N64 without on-the-fly conversion.
OS: Debian GNU/Linux Jessie (8.0)
CPU: Intel i7 4770K @ 3.5 GHz
Build: AVX (compiled from git)

User avatar
wareya
Posts: 16
Joined: Tue May 19, 2015 5:44 pm

Re: Support .n64

Post by wareya » Sun Oct 11, 2015 6:40 am

Snowstorm64 wrote: Well said. I would add that although it looks a trivial thing, it is not; because they have done a serious disservice to the N64's preservation by uploading and spreading byteordered ROMs that aren't the very same ones that are found in the N64 cartridges, and that those ROMs wouldn't boot on a real N64 without on-the-fly conversion.
Added a semicolon for fluid grammar. Took a while for me to unnest those clauses.

MarathonMan, fight that good fight ;)

User avatar
DaFox
Posts: 1
Joined: Wed Apr 08, 2015 10:28 pm

Re: Support .n64

Post by DaFox » Fri Oct 16, 2015 7:43 am

MarathonMan wrote:N64's history is filled to the brim with hacks layered on top of hacks and I won't contribute to or tolerance such nonsense.

There's simply no reason for more than one format to exist, and I believe big-endian is the "right" one, and as an emulator author I choose to use my position to advocate for the adoption of one file format.
I've only just woken up but this is the best thing I've heard today so far.

User avatar
Narann
Posts: 154
Joined: Mon Jun 16, 2014 4:25 pm
Contact:

Re: Support .n64

Post by Narann » Mon Oct 19, 2015 1:30 pm

MarathonMan wrote:N64's history is filled to the brim with hacks layered on top of hacks and I won't contribute to or tolerance such nonsense.
Writing a video plugin for fun, I couldn't agree more.

The more I'm in N64 emu scene, the more I want to rewrite everything...

User avatar
Snowstorm64
Posts: 303
Joined: Sun Oct 20, 2013 8:22 pm

Re: Support .n64

Post by Snowstorm64 » Wed Jan 27, 2016 8:53 pm

I'm bumping this thread because I have just discovered that SRAM files, when created on any emulator other than CEN64, are byteswapped (I'm not sure about EEPROM and FlashRAM yet), I say so because I have downloaded them from Internet, and some SRAM won't load on CEN64, but they'll do on mupen64plus, and vice versa. This is annoying.
OS: Debian GNU/Linux Jessie (8.0)
CPU: Intel i7 4770K @ 3.5 GHz
Build: AVX (compiled from git)

User avatar
MarathonMan
Site Admin
Posts: 692
Joined: Fri Oct 04, 2013 4:49 pm

Re: Support .n64

Post by MarathonMan » Thu Jan 28, 2016 12:33 am

If other emulators are using byteswapped save files, I would file a bug report with those emulators. ;)

The N64 scene is broken, and has been for a very long time. I think you'll agree with me when I say that practically any message board on the internet will tell you that the N64 is probably one of the most poorly emulated consoles to date.

I'm not going to follow in the footsteps of those who continue to pile more hacks on top of hacks with millions of configuration boxes and byte-formatted ROMs and save files.

</rant>

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest