Compatibility Database

Discuss ROM-compatibility related issues here.
Post Reply
beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Compatibility Database

Post by beannaich » Thu Jan 02, 2014 5:04 pm

Hello,

I have a basic compatibility database schema that I employed for my private emulator project, and would like to provide it to you so a proper database could be set up and maintained. :D

Since I do this sort of thing for a living, I could also build out a web interface so users can interact with and make changes to said database.

Let me know if you're interested and we can talk about what to do moving forward.

User avatar
MarathonMan
Site Admin
Posts: 692
Joined: Fri Oct 04, 2013 4:49 pm

Re: Compatibility Database

Post by MarathonMan » Fri Jan 03, 2014 10:22 am

beannaich wrote:Hello,

I have a basic compatibility database schema that I employed for my private emulator project, and would like to provide it to you so a proper database could be set up and maintained. :D

Since I do this sort of thing for a living, I could also build out a web interface so users can interact with and make changes to said database.

Let me know if you're interested and we can talk about what to do moving forward.
I think it's a great idea, though I'm not sure that I would want to host it here (though I'd gladly link to it and support it). The VM that's running this website is a bargain-bin special and doesn't really have memory for anything more than a simple phpBB instance. I'm also constantly under attack and don't have the time to audit another website...

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Fri Jan 03, 2014 10:34 pm

MarathonMan wrote:I think it's a great idea, though I'm not sure that I would want to host it here (though I'd gladly link to it and support it). The VM that's running this website is a bargain-bin special and doesn't really have memory for anything more than a simple phpBB instance. I'm also constantly under attack and don't have the time to audit another website...
That's a shame, this schema is all ready to go, maybe next week I'll look into hosting and all that fun stuff.

EDIT: Here's the structure. It's worked pretty well for me so far, I'll see about getting it online for you.

Image

User avatar
MarathonMan
Site Admin
Posts: 692
Joined: Fri Oct 04, 2013 4:49 pm

Re: Compatibility Database

Post by MarathonMan » Sat Jan 04, 2014 1:31 am

Schema is only the sword draw in a battle against those looking to exploit you. :)

I'd love to provide comments (if any), but I'm not familiar with M$ SQL design tools (at least, that's what it looks like it is). Can you get a dump of the schema grammar in plaintext form?

User avatar
Snowstorm64
Posts: 303
Joined: Sun Oct 20, 2013 8:22 pm

Re: Compatibility Database

Post by Snowstorm64 » Sat Jan 04, 2014 8:04 am

I think we should avoid any Microsoft's solution. I suggest SQLite, it is a very lightweight SQL database that can also be embedded into CEN64 or in a website.

I think SQLite is enough for purposes like the compatibility list.

See the website for further information: http://www.sqlite.org/
OS: Debian GNU/Linux Jessie (8.0)
CPU: Intel i7 4770K @ 3.5 GHz
Build: AVX (compiled from git)

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Sat Jan 04, 2014 1:07 pm

Snowstorm64 wrote:I think we should avoid any Microsoft's solution. I suggest SQLite, it is a very lightweight SQL database that can also be embedded into CEN64 or in a website.

I think SQLite is enough for purposes like the compatibility list.

See the website for further information: http://www.sqlite.org/
I'm almost positive you can query a MSSQL database from Linux/Mac/Whatever. Also, I'm sure MarathonMan would like compatibility database stuff in the front ends, and not have to deal with it directly. It at least makes the most sense to do it that way. I'll have my own website for my emulation project, and I'll be able to host the database there in the interim. If at some point in the future someone would like to take over that responsibility, we can easily generate scripts to transfer data to their setup.

User avatar
MarathonMan
Site Admin
Posts: 692
Joined: Fri Oct 04, 2013 4:49 pm

Re: Compatibility Database

Post by MarathonMan » Sat Jan 04, 2014 1:10 pm

M$ SQL works from any type of connection. It's just an implementation of a SQL database; all that's required is for both parties to speak SQL.
beannaich wrote:Also, I'm sure MarathonMan would like compatibility database stuff in the front ends, and not have to deal with it directly.
Oh my, yes. A database has no business in the backend.

User avatar
Breadwinka
Posts: 54
Joined: Fri Oct 04, 2013 11:35 pm

Re: Compatibility Database

Post by Breadwinka » Sat Jan 04, 2014 2:06 pm

I could host the database if no one wants to.

User avatar
Snowstorm64
Posts: 303
Joined: Sun Oct 20, 2013 8:22 pm

Re: Compatibility Database

Post by Snowstorm64 » Sat Jan 04, 2014 3:52 pm

Well, if you have good technical reasons to not use SQLite, I won't complain, I thought it was a good idea because it is lightweight and, if we wish, it can be embedded into any CEN64 frontend so those can load Good names, checksums, ratings, CRC, etc.. But... should we really use MS-SQL or any solution made by Microsoft? The idea of using proprietary stuff makes me feel uncomfortable...I won't object to it if you insist, but I think there are better alternatives to it.
OS: Debian GNU/Linux Jessie (8.0)
CPU: Intel i7 4770K @ 3.5 GHz
Build: AVX (compiled from git)

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Sat Jan 04, 2014 4:40 pm

Snowstorm64 wrote:Well, if you have good technical reasons to not use SQLite, I won't complain, I thought it was a good idea because it is lightweight and, if we wish, it can be embedded into any CEN64 frontend so those can load Good names, checksums, ratings, CRC, etc.. But... should we really use MS-SQL or any solution made by Microsoft? The idea of using proprietary stuff makes me feel uncomfortable...I won't object to it if you insist, but I think there are better alternatives to it.
It's a version of SQL that I'm familiar with, and I won't be supporting good names either. Goodsets have bad dumps and ROM hacks in them, it'll be more like Dolphin/PCSX2's database. If a different version of a game exists it could be added, but the database has explicit region definitions (USA, Japan, Europe for now), so the good name conventions (, [J], [E] etc) aren't necessary. Publisher/Developer information is also supplied for the (exceedingly rare) instances where two companies make a game with the same name, but are different binaries. AFAIK, that only happened in the NES era, but it'd be handy to have anyway. Release date is also supplied for the same reason. Software version is also supplied so regressions can be more easily tracked from one version of the emulator to another.

Is there any distinguishing information that anyone thinks is necessary that isn't known from a game's name/file name alone? I don't necessarily want to turn this database into a history of gaming, so superfluous information (Box art, number of players, instruction manuals, etc) probably won't make it in.

User avatar
Snowstorm64
Posts: 303
Joined: Sun Oct 20, 2013 8:22 pm

Re: Compatibility Database

Post by Snowstorm64 » Sat Jan 04, 2014 5:16 pm

beannaich wrote: It's a version of SQL that I'm familiar with, and I won't be supporting good names either. Goodsets have bad dumps and ROM hacks in them, it'll be more like Dolphin/PCSX2's database. If a different version of a game exists it could be added, but the database has explicit region definitions (USA, Japan, Europe for now), so the good name conventions (, [J], [E] etc) aren't necessary. Publisher/Developer information is also supplied for the (exceedingly rare) instances where two companies make a game with the same name, but are different binaries. AFAIK, that only happened in the NES era, but it'd be handy to have anyway. Release date is also supplied for the same reason. Software version is also supplied so regressions can be more easily tracked from one version of the emulator to another.

Is there any distinguishing information that anyone thinks is necessary that isn't known from a game's name/file name alone? I don't necessarily want to turn this database into a history of gaming, so superfluous information (Box art, number of players, instruction manuals, etc) probably won't make it in.


I liked Good Name conventions for obvious reasons, of course I don't mean to support the goodsets, I only support No Intro sets. :) But what have you done is excellent and I'm fine with that. ;)

I think md5 checksums should also be there, so one could see if a ROM is perfectly dumped or not. Save types (EEPROM, SRAM, FlashRAM, Controller Pack) are essential for CEN64, because if you don't use the right format, the savegame won't work. CRC1 and 2 could be useful for cheats, but I'm not sure if we really need that. I don't think we need anything other.
OS: Debian GNU/Linux Jessie (8.0)
CPU: Intel i7 4770K @ 3.5 GHz
Build: AVX (compiled from git)

User avatar
rogerhanin2002
Posts: 7
Joined: Sun Oct 20, 2013 7:35 am

Re: Compatibility Database

Post by rogerhanin2002 » Sat Jan 04, 2014 6:50 pm

beannaich wrote: Is there any distinguishing information that anyone thinks is necessary that isn't known from a game's name/file name alone? I don't necessarily want to turn this database into a history of gaming, so superfluous information (Box art, number of players, instruction manuals, etc) probably won't make it in.
save format for sure, but also rumble pak, controler pak, expansion pak, transfer pak, microphone, and perhaps future hypothetic 64dd
and multiplayer mode, high resolution mode, widescreen mode.
a good start for database could be :
http://www.nesworld.com/n64releaselist.php
or
http://64dd.net/modules/games/index.php ... &order=asc
maybe ask them for some help.

I like the idea of save files to.

ps: i have no interest in any of those website

User avatar
Mizox
Posts: 17
Joined: Fri Oct 04, 2013 8:24 pm

Re: Compatibility Database

Post by Mizox » Sat Jan 04, 2014 8:42 pm

rogerhanin2002 wrote: save format for sure, but also rumble pak, controler pak, expansion pak, transfer pak, microphone, and perhaps future hypothetic 64dd
and multiplayer mode, high resolution mode, widescreen mode.
don't forget that there's also a couple different variations on widescreen mode.

for example... Donkey Kong 64 and Banjo-Tooie had anamorphic Widescreen (most of the time anyway. DK64's would turn off during cutscenes)

whereas say... Road Rash 64 used letterbox widescreen

and Goldeneye 007 supported both methods I think

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Sun Jan 05, 2014 2:00 pm

Snowstorm64 wrote:I think md5 checksums should also be there, so one could see if a ROM is perfectly dumped or not. Save types (EEPROM, SRAM, FlashRAM, Controller Pack) are essential for CEN64, because if you don't use the right format, the savegame won't work. CRC1 and 2 could be useful for cheats, but I'm not sure if we really need that. I don't think we need anything other.
Back-up type could definitely be supported, since the front-ends could just look it up and pass it as a parameter to cen64. I won't be supporting cheats though, this database is all about compatibility status.
rogerhanin2002 wrote:save format for sure, but also rumble pak, controler pak, expansion pak, transfer pak, microphone, and perhaps future hypothetic 64dd
and multiplayer mode, high resolution mode, widescreen mode.
I suppose which peripherals are supported by a title could be useful ("The controller pak appears to be damaged") though I think it would be better to let front-ends handle this. In reality, there's nothing stopping you from plugging a rumble pak into your controller and playing a game that doesn't support it, some games may also do weird things on a real console if you do that, and I'd love for that to be preserved as well. High-res/Widescreen/Multi-player probably won't make it in.

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Thu Jan 09, 2014 9:50 pm

Apologies for the double post, I'll be getting web hosting tomorrow for my personal project, and will get the CEN64 database up for testing this weekend.

Not sure what MarathonMan's schedule is like for the next few days though. Either way, it should be up for testing by no later than Monday*, I'll also need to hear from users who would like access to the database and are willing to add records/help maintain the integrity of the data.

Cheers

*subject to change without notice

User avatar
MarathonMan
Site Admin
Posts: 692
Joined: Fri Oct 04, 2013 4:49 pm

Re: Compatibility Database

Post by MarathonMan » Thu Jan 09, 2014 10:44 pm

beannaich wrote:*subject to change without notice
You monster!

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Sat Jan 11, 2014 12:13 pm

Should be getting hosting today, database will be up as soon as possible for testing.

User avatar
Snowstorm64
Posts: 303
Joined: Sun Oct 20, 2013 8:22 pm

Re: Compatibility Database

Post by Snowstorm64 » Sun Feb 02, 2014 1:10 pm

When will we see this database?
OS: Debian GNU/Linux Jessie (8.0)
CPU: Intel i7 4770K @ 3.5 GHz
Build: AVX (compiled from git)

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Sun Feb 23, 2014 10:38 pm

Snowstorm64 wrote:When will we see this database?
I ran into a few setbacks with it, as well as just being busy with work and my personal life. I'm building a front end for it right now, should be done soon. :)

Image

That's the current table layout for the database, I keep going back and forth on what kind of front end I want for it (Which isn't relevant for CEN64, since MM or someone else will have to make a CEN64-centric one). But I need one for testing the database, so that's really what's taking so long. Keep in mind, I'm developing this database for my own emulation project, and I like to take my time and make sure things are perfect. :)

If need be, the database SHOULD be in a usable state, so once I get hosting I could throw it somewhere with a crappy default MVC 5 front end.

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Thu Oct 02, 2014 8:02 pm

The database generation scripts have been sent to marathonman, and I am building the appropriate web interfaces to manipulate the data and view it in various, helpful forms.

One view I imagine will show all games for a particular version that were never rated for compatibility. This will hopefully give users a better vantage point to look for compatibility gaps.

I'm also looking into making a compatibility checker daemon that will be able to use various metrics to determine not only if a game is working or not, but what its compatibility rating is (broadly speaking, it can't check for audio/video errors, that would require way too much work). Possibly even enable smart commit messages so if a commit fixes a certain game, you could tell the daemon to only run checks for that title. :)

User avatar
Nintendo Maniac 64
Posts: 185
Joined: Fri Oct 04, 2013 11:37 pm

Re: Compatibility Database

Post by Nintendo Maniac 64 » Tue Jan 13, 2015 2:19 am

Any update on this? With the recent developments this could be really handy...
CEN64 Forum's resident straight-male kuutsundere
(just "tsundere" makes people think of "Shana clones" *shivers*)

CPU+iGPU: Pentium G3258 @ 4.6GHz/1.281v
dGPU: Radeon HD5870 1GB
RAM: Vengeance 1600 4x4GB
OS: Windows 7

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Wed Jan 14, 2015 1:12 am

Nintendo Maniac 64 wrote:Any update on this? With the recent developments this could be really handy...
Very busy with work. The schema was sent to MarathonMan and he ran the scripts against a MySQL instance. So the tables exist and I have some initial attempts at a web front end for accessing and adding data. If no one with web development experience comes forward, it'll have to wait until I get free time.

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Fri Jan 16, 2015 12:50 pm

Excuse the double post. I have a three day weekend this weekend, so I may be able to find some free time to work on the compatibility pages.

Not a guarantee though.

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Sat Mar 28, 2015 12:11 pm

Got some more time to work on this stuff recently.

Image

Pretty bare bones right now, but a lot of the infrastructure is now in place.

User avatar
Snowstorm64
Posts: 303
Joined: Sun Oct 20, 2013 8:22 pm

Re: Compatibility Database

Post by Snowstorm64 » Mon Mar 30, 2015 11:03 am

Nice! A simple, clean layout is the way to go with it. Do you have an ETA of when will it be uploaded on the site?
OS: Debian GNU/Linux Jessie (8.0)
CPU: Intel i7 4770K @ 3.5 GHz
Build: AVX (compiled from git)

User avatar
chriztr
Posts: 38
Joined: Sun Oct 06, 2013 4:15 pm

Re: Compatibility Database

Post by chriztr » Mon Mar 30, 2015 4:59 pm

>like

User avatar
Nintendo Maniac 64
Posts: 185
Joined: Fri Oct 04, 2013 11:37 pm

Re: Compatibility Database

Post by Nintendo Maniac 64 » Wed Apr 01, 2015 3:20 am

It may be wise to be able to specify the OS as well since some issues are not universal between platforms.
CEN64 Forum's resident straight-male kuutsundere
(just "tsundere" makes people think of "Shana clones" *shivers*)

CPU+iGPU: Pentium G3258 @ 4.6GHz/1.281v
dGPU: Radeon HD5870 1GB
RAM: Vengeance 1600 4x4GB
OS: Windows 7

User avatar
Narann
Posts: 154
Joined: Mon Jun 16, 2014 4:25 pm
Contact:

Re: Compatibility Database

Post by Narann » Fri Apr 03, 2015 4:44 pm

Great!

beannaich
Posts: 149
Joined: Mon Oct 21, 2013 2:43 pm

Re: Compatibility Database

Post by beannaich » Sun Apr 05, 2015 2:54 am

Snowstorm64 wrote:Nice! A simple, clean layout is the way to go with it. Do you have an ETA of when will it be uploaded on the site?
I agree that simple, and clean is the way to go. Mostly because I'm not artistically inclined :D

No ETA, although I could upload the title bar and a basic home page tomorrow when I get the chance.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest