Support NeoGAF

[Gestault]

Disgusting. Taking 'pictures' of stuff is hardly comparable .

Here's what I do when I scan album booklets for preservation:

If we're going to make a show of our processes, I'm surprised you don't convert to LAB color so you can run the Unsharp Mask without it hitting the color channels, then converting back.

It's not my process, but I've seen some legitimately stunning setups for photography in place of traditional scanning for 2D art. It's not like getting a webcam and standing over a book opened spine-out. There's math specific to your lens in the finishing stages, and proper racking for everything. If memory serves, some of the New York Public Library historical preservation efforts used that process for their Jan 2016 digital collection.

 

[Dio]

Consider me suitable reprimanded. Is it really a big difference? Assuming similar flat image and even 45 degree lighting to prevent glare? Just seems like scanners haven't moved on much in 10 years when digital cameras have.

Yes, it's a really big difference.

You don't want any variable lighting at all. The idea is to create a scanned image that, if you printed it onto a piece of paper, would probably be almost identical to the original cover for the game. Maybe I'm neurotic but the presence of shade and lighting and glare and dust and all that is really easy to spot and near impossible to get rid of in a photograph.

If we're going to make a show of our processes, I'm surprised you don't convert to LAB color so you can run the Unsharp Mask without it hitting the color channels, then converting back.

It's not my process, but I've seen some legitimately stunning setups for photography in place of traditional scanning for 2D art. It's not like getting a webcam and standing over a book opened spine-out. There's math specific to your lens in the finishing stages, and proper racking for everything. If memory serves, some of the New York Public Library historical preservation efforts used that process for their Jan 2016 digital collection.

True, but at that point you're expending MORE effort than scanning. The perfectly flat, uniform lighting look is really hard to get right without creating a whole setup for it.

 

[foltzie1]

It's common to underestimate the work involved.

First step is to clean the carts. I turn this:

Into this:

This step takes about 2-30 minutes.

Then I open the cart, and clean the contacts to ensure a good dump the first try. This takes 3-5 minutes to clean both sides.

Then I scan the PCB and game cartridge at 600DPI. This takes about 5 minutes (my scanner's a bit slow.) Then where possible, I use a heat gun to peel off stickers from boxes, takes about 2-3 minutes. Then scan the box front and back. Each takes 5 minutes.

Then I dump the cartridge using a serial connection between my PC and the SNES. This used to take 20-30 minutes when I was doing it over the controller port with the USA set, but it now takes 5-10 minutes thanks to an expansion port device made for me by defparam.

Then I log all the serial#s on the board and cartridge into a large database by hand. This takes another 2-3 minutes.

Rinse, repeat for every game.

I have very low energy levels and become exhausted after about 2-4 hours of this a day. I've had a lot of theories but never found the definitive cause. But my current guess is that I might have very low testosterone levels. But it's something. Because damn, I am always, always tired no matter how much I sleep.

Quite impressive, please excuse my ponderings as the ponderings they are. I was of course only thinking of the dumps as the largest portion of the work. Clearly that was mistaken.

If I may add something of value, perhaps a call into the local media may be of value w/r the missing games. The local station managers malaise my suddenly turn around if he starts getting questions from the media.

Also, contacting the USPS Inspector Generals Office may be of some value: http://usa-faq.civicagency.org/answers/9690

 

[chronic_archaic]

Fuck, I forgot who byuu was - holy shit dude I hope things get better.

 

[Gestault]

True, but at that point you're expending MORE effort than scanning. The perfectly flat, uniform lighting look is really hard to get right without creating a whole setup for it.

Yeah, for the topic here, it would end up being more work for the person unless it was already their specialty. It exists in part because some items/manuscripts CAN'T be flat-pressed the way you'd want for normal scanning.

 

[EmiPrime]

Hey everyone, thanks for the concern and well wishes.

I noticed quite a few questions regarding insurance, why I'm doing this, and the like, so I wrote an article to try and answer everyone's questions. I hope this helps.

https://byuu.org/emulation/preservation/lost-package/

If anyone reads the entirety of the article (it's lengthy, sorry), and still has questions, I'll be happy to answer them.

Thank you all!

All the best byuu, I hope the package turns up. Thank you for your (considerable) contributions to game preservation.

 

[LeleSocho]

This situation enrages me, all postal services are scum on earth and i dread the times i'm forced to use them.

 

[Kagoshima_Luke]

I'm curious as to what he means by "bad dumps." Does that mean info on the source cartridge was somehow damaged, sort of like disc rot with cds/dvds?

 

[Robin64]

I'm curious as to what he means by "bad dumps." Does that mean info on the source cartridge was somehow damaged, sort of like disc rot with cds/dvds?

From here:

Second, there are bad dumps out there. There are many reasons for these. One is that games were often patched to remove anti-copier protections. These often do serious things like slow games down by up to 25% of their original speed (though usually it's not that drastic), because the oldest copiers did not have RAM with fast access speeds inside of them. Another is that due to the use of floppy disks, bits would get flipped occasionally. The third would be from older piracy groups adding "trainers" (advertisements upon booting the game, often with the ability to apply cheats to the game from an onscreen menu), and sometimes people would remove these trainers rather than redumping the games. The fourth would be header changes to make games run in emulators with poor heuristics. And the fifth and least likely would be malicious changes: people putting their names into the game images for bragging rights. Most notably here would be Diskdude and Vimm's Lair.

In the first batch of 100 PAL games I dumped, I found two games with bit corruption. The first was Spider-Man & Venom, where the main Spider-Man sprite was partially corrupted. The second was Fatal Fury 2, where one of the fighter sprite frames was partially corrupted. What's so damning about this is that both of these games were marked as "verified" in GoodSNES, which for many was considered a gold standard that the games were 100% bit-perfect copies.

A friend, KingMike, has found a half-dozen bad dumps of Japanese games from his own collection so far.

 

[Fisty]

Wouldn't it have been easier, cheaper, and safer to just take a couple days/weeks and fly over to Germany and do the dumps there? Seems like a really poorly thought out plan

 

[goodcow]

If I sell food to a billion people, and 0.001% of them get food poisoning and die, it doesn't really matter if I'm doing a great job statistically, because I still killed 10,000 people.

Granted, this isn't quite a life or death situation, but I don't think it's unreasonable to expect mail carriers to actually deliver a valuable package, unless you're using a dirt cheap service option.

So you're telling me you don't fuck up at least 0.001% of the time at work? Mistakes happen. That's why there's insurance and options like registered mail. Nobody is perfect.

Edit: a friend hand made me crocheted Sega characters in Little Big Planet style. I paid for registered mail when she shipped them to me. Why? Because they were irreplaceable.

 

[goodcow]

US postal service sounds terrible! Although I remember UPS "losing" one of our trackers we posted to the USA and they found it almost the same day we told them it was a GPS tracker

Except it's not. I get at least 200 parcels a year, and nothing has ever been lost by the USPS in over a decade. Twice, things have been missing for a few weeks, but they eventually showed up.

Edit: Meanwhile, back when DHL actually existed in America, they stole a digital camera I ordered on Amazon. It showed delivered, but the security camera footage in the building showed the driver coming to the door and just leaving. Also had Lasership parcels go missing. Nothing stolen/lost by UPS or FedEx, but damaged shipments.

 

[Head.spawn]

Is he the one who created the very accurate Bsnes emulator?

Yessir.

 

[nkarafo]

From here:

I was under the impression that roms with the (!) indicator are confirmed good dumps.

Although i admit that i'm not sure if all SNES PAL roms have this. US roms do however.

 

[Spiral Insanity]

I just want to give thanks to people like byuu who are so important to the preservation of gaming history. Amazing dedication and talent. I hope this gets sorted out.

 

[Grassy]

He needs to be tweeting this at Nintendo. With stories like this that make even a bit of publicity, usually someone will get off their ass and find it.

Do you really think Nintendo would give two fucks about a guy who created a SNES emulator not receiving the SNES cartridges whose Rom files he was going to dump?

 

[ScientificPizza]

Sending 10k$ worth of games with standard mail? Pay some more and use a courier, jesus. This sucks, but it could have been avoided.

Yeah, it just seems poorly planned if anything

 

[Wowfunhappy]

So you're telling me you don't fuck up at least 0.001% of the time at work? Mistakes happen.

Oh, I screw up way more often than that. And then a coworker catches it, because there's redundency built into the system.

 

[Mandoric]

I was under the impression that roms with the (!) indicator are confirmed good dumps.

Although i admit that i'm not sure if all SNES PAL roms have this. US roms do however.

This project is to audit those while adding technical details and good packaging scans.

Most have been right, but some haven't. (Or, I guess to be accurate, byuu and KingMike have each ended up with rips which differ from the goodset but are reproducable and solve errors in human-readable data.)

In general, Cowering's team seems to have done purely digital forensics (at least on rarer titles) and marked things as "known good" if all internet copies agreed/there were no visible signs of modification, which is of course vulnerable to both situations where there was a single dump and situations where multiple dumps were bad in different ways.

That's why rerips, from multiple carts, are so important. You can't just apply forensic techniques, even combined with replays, or you'll end up accepting false "goods" like the ones they've found, while potentially rejecting false "bads" like FF6 with its single-bit shuffle that removes the evade stat from the game.

 

[SScorpio]

Yeah, it just seems poorly planned if anything

Things way more expensive than that have been shipped via USPS. Like I don't know, the Hope Diamond. https://about.usps.com/who-we-are/postal-history/hope-diamond.pdf

But in all seriousness, USPS is one of, if not the mail system in the entire world. Sometimes stuff does happen, but it's down to local issues which normally get flushed out when things like this happen.

 

[byuu]

I was under the impression that roms with the (!) indicator are confirmed good dumps.

That's what it's supposed to mean. But there's no proper validation process. It doesn't say who dumped the game, how they dumped it, and it doesn't require any proof (in the form of a 600dpi scan of the cartridge and PCB.) So we don't even know if the cart they dumped was even legitimate or a bootleg.

Example, GoodSNES says:
Spider-Man & Venom – Maximum Carnage (E) [!]

And yet, its copy is damaged. As is the copy in No-Intro. See:
https://forums.nesdev.com/viewtopic.php?f=12&t=14799

Here's another damaged game, Fatal Fury. Although I don't believe this one was marked verified.
https://forums.nesdev.com/viewtopic.php?f=12&t=14920

 

[DiipuSurotu]

Wouldn't it have been easier, cheaper, and safer to just take a couple days/weeks and fly over to Germany and do the dumps there?

Found some answers to some of the questions in this thread:

.

 

[Zaraki_Kenpachi]

That's what it's supposed to mean. But there's no proper validation process. It doesn't say who dumped the game, how they dumped it, and it doesn't require any proof (in the form of a 600dpi scan of the cartridge and PCB.) So we don't even know if the cart they dumped was even legitimate or a bootleg.

Example, GoodSNES says:
Spider-Man & Venom – Maximum Carnage (E) [!]

And yet, its copy is damaged. As is the copy in No-Intro. See:
https://forums.nesdev.com/viewtopic.php?f=12&t=14799

Here's another damaged game, Fatal Fury. Although I don't believe this one was marked verified.
https://forums.nesdev.com/viewtopic.php?f=12&t=14920

I literally can't even see the difference in the Spiderman one....

 

[Spiral Insanity]

I literally can't even see the difference in the Spiderman one....

It's right there on his leg, gray line.

 

[TLZ]

"Lost".

 

[BigEmil]

"Lost".

Definitely stolen in the process within the postal service

 

[Muninn]

Someone stole those games.

Yeah what is going on lol. This is stuff you package with premium shipping services. What on earth happened here?

 

[Madness]

You have to wonder why billion dollar companies like Nintendo wouldn't do something like this. It is always the amateur fans who do it. Sucks for the collector. I didn't know Mega Man X series was so rare. I have all 3 in my storage. Wonder what I could get for them.

 

[BumblebeeCody]

I just want to give thanks to people like byuu who are so important to the preservation of gaming history. Amazing dedication and talent. I hope this gets sorted out.

My thoughts exactly.

 

[DiipuSurotu]

You have to wonder why billion dollar companies like Nintendo wouldn't do something like this.

Because they're a for-profit business? Why would they do something like this?

I mean I totally support what byuu is doing but the answer to your question should be obvious.

 

[Lynd7]

So is he trying to do all regions?

 

[Decider]

Because they're a for-profit business? Why would they do something like this?

I mean I totally support what byuu is doing but the answer to your question should be obvious.

Not really. Expecting that Nintendo would keep their own digital archive of their games for retail via VC at the very least isn't exactly out of the ordinary.

 

[Spectone]

Losing all these cartridges is just heartbreaking, stupid USPS.

 

[Plumpbiscuit]

Stolen imo, you don't just "lose" over 100 cartridge games just like that, especially given the weight and size of the parcel it would probably be too much of an eye sore to simply lose. Somebody was curious, broke into the box, saw the gold, took it home. He should have sent it with insurance at the very least.

 

[Bendo]

So is he trying to do all regions?

Yes. He collected and dumped/scanned the entire US set, which he then sold to purchase the entire JP set. The PAL set is, from what I understand, incredibly expensive and hard to find, which is why he opted to borrow it instead.

Also, I wish people would stop with all the knee-jerk "lol, why not go to germany", or "insure it better dummy". It's already been asked and answered multiple times over, and adds nothing.

 

[petran79]

Hey everyone, thanks for the concern and well wishes.

I noticed quite a few questions regarding insurance, why I'm doing this, and the like, so I wrote an article to try and answer everyone's questions. I hope this helps.

https://byuu.org/emulation/preservation/lost-package/

If anyone reads the entirety of the article (it's lengthy, sorry), and still has questions, I'll be happy to answer them.

Thank you all!

Is travelling to the USPS warehouse in Jersey an option, especially for such a valuable package?

Speaking to the manager there directly and give contact info to employees might be more effective than emails, phone calls or involving third parties

 

[Cheerilee]

I was under the impression that roms with the (!) indicator are confirmed good dumps.

Although i admit that i'm not sure if all SNES PAL roms have this. US roms do however.

SNES games were originally dumped by dozens/hundreds of different people, all using different copiers and different standards. Filenames would frequently get changed and ROM collectors couldn't tell if they had discovered a new ROM or if someone just didn't like the way the old one was named. When something like a translation patch came out, ROMs were frequently redistributed by people who thought they were doing the world a favor by patching a ROM and floating a pre-patched version into the wild.

The "GoodSNES" project tried to gather and sort that disorganized mess into good ROMs and bad ROMs.

They tested the ROMs by loading them into emulators (probably ZSNES, which is not a perfectly accurate emulator) and making a quick judgement on whether the ROM is good or bad. GoodSNES never actually re-dumped the games and looked to see if they were digitally perfect, they just played them to see if they played okay. That's how you get errors like those grey pixels on Spiderman's leg, with people saying "I didn't even notice something so small."

byuu personally redumping every single game and ensuring that each dump is digitally perfect is much more accurate than GoodSNES.


And on top of that, byuu is introducing a new standard for dumping. Many large SNES games create one big ROM by using two or more smaller ROM chips. The old dumping standard just smooshed the chips together and called it one big ROM, but that's inaccurate and it triggers things like the Earthbound copy protection. A truly accurate dump of a game should say so when one ROM is made out of two ROMs.

 

[Charles Brandon]

so...er...this perfect set of US ROM dumps that byuu has...any chance of...er...borrowing them?

 

[Cheerilee]

so...er...this perfect set of US ROM dumps that byuu has...any chance of...er...borrowing them?

He's already said that's never going to happen. He would prefer not to be arrested for piracy.

But he is making the verification info available for his perfect copies. And defparam is working on trying to make an inexpensive version of byuu's perfect copier. That means that eventually you will be able to buy a perfect copier and dump your own perfect games, and you'll be able to confirm against byuu's mega-collection that your own mini-collection really is perfect.

If anonymous strangers on the internet choose to push their luck with the authorities and start trading perfect copies amongst themselves until a full-sized perfect collection is easily piratable, that's somebody else's concern. And it shouldn't be too much of a concern, IMO, because GoodSNES exists. The SNES piracy bird has already flown.

 

[Charles Brandon]

Gotcha!

 

[sixteen-bit]

"Lost".

Pretty much, lol

 

[MooMilk2929]

It's only been in the US 11 days. It's too early to declare it lost. I've had packages from the US take a while too. And sometimes it isn't scanned during the trip until it's arrives at your post office. You need to wait longer if you are going to have a fundraiser I think. Especially since you think you might not be able to give refunds. The guy who sent it can wait a while too cause it's not like he was about to sell them so he's missing money.

 

[nkarafo]

SNES games were originally dumped by dozens/hundreds of different people, all using different copiers and different standards. Filenames would frequently get changed and ROM collectors couldn't tell if they had discovered a new ROM or if someone just didn't like the way the old one was named. When something like a translation patch came out, ROMs were frequently redistributed by people who thought they were doing the world a favor by patching a ROM and floating a pre-patched version into the wild.

The "GoodSNES" project tried to gather and sort that disorganized mess into good ROMs and bad ROMs.

They tested the ROMs by loading them into emulators (probably ZSNES, which is not a perfectly accurate emulator) and making a quick judgement on whether the ROM is good or bad. GoodSNES never actually re-dumped the games and looked to see if they were digitally perfect, they just played them to see if they played okay. That's how you get errors like those grey pixels on Spiderman's leg, with people saying "I didn't even notice something so small."

byuu personally redumping every single game and ensuring that each dump is digitally perfect is much more accurate than GoodSNES.


And on top of that, byuu is introducing a new standard for dumping. Many large SNES games create one big ROM by using two or more smaller ROM chips. The old dumping standard just smooshed the chips together and called it one big ROM, but that's inaccurate and it triggers things like the Earthbound copy protection. A truly accurate dump of a game should say so when one ROM is made out of two ROMs.

Ok got it.

What about other scene releases though? Like No-Intro for instance? Are these the same roms? Did they re-dumped them? Are they good?

 

[Marty Chinn]

He's already said that's never going to happen. He would prefer not to be arrested for piracy.

But he is making the verification info available for his perfect copies. And defparam is working on trying to make an inexpensive version of byuu's perfect copier. That means that eventually you will be able to buy a perfect copier and dump your own perfect games, and you'll be able to confirm against byuu's mega-collection that your own mini-collection really is perfect.

If anonymous strangers on the internet choose to push their luck with the authorities and start trading perfect copies amongst themselves until a full-sized perfect collection is easily piratable, that's somebody else's concern. And it shouldn't be too much of a concern, IMO, because GoodSNES exists. The SNES piracy bird has already flown.

What does he plan to do with a perfect archive if nobody can access it?

 

[alr1ght]

Ok got it.

What about other scene releases though? Like No-Intro for instance? Are these the same roms? Did they re-dumped them? Are they good?

Back in the day, people ripped Roms and put their group logo at the beginning. No-Intro was started to get clean rips without any added junk, hence the name.

 

[Cow Mengde]

I was under the impression that roms with the (!) indicator are confirmed good dumps.

Although i admit that i'm not sure if all SNES PAL roms have this. US roms do however.

Same. I always run SNES9x to see if it says good checksum on my roms.

 

[nkarafo]

Back in the day, people ripped Roms and put their group logo at the beginning. No-Intro was started to get clean rips without any added junk, hence the name.

I see.

They are still seem to update their roms so i was wondering if they also do re-dumps to ensure good quality.

 

[byuu]

so...er...this perfect set of US ROM dumps that byuu has...any chance of...er...borrowing them?

No, and it's not necessary.

The SHA256 sums that I have already released can be used to validate that your ROMs are correct. There is no known way to create SHA-1 collisions, let alone for the much more secure SHA-256.

If your SHA256 matches mine, there's a 100% chance right now that it's the bit-perfect original ROM.

When it comes to simple bit-corruptions, I just listed the corrupted bytes since it's only been a couple bytes so far. That has allowed people to fix their copies and update their databases, so they'll probably propagate in future ROM sets. But I'm not in any way involved with those guys, so I don't know.

You need to wait longer if you are going to have a fundraiser I think.

Oh, definitely!! I'm not going to ask for money until we're at least three months after shipping, so April at the very earliest. I don't want to receive money, and then receive the games later on.

Basically, everyone on r/emulation was saying I should use fundraising to help repay the guy for the lost games, and I said I'd strongly consider that. Although the fees those sites charge (8%!!) is outright criminal. Here I thought Paypal's 3% was too high.

No-Intro was started to get clean rips without any added junk, hence the name.

My biggest issue with No-Intro is that they're pragmatists. When they don't have a clean ROM, they just index an unclean one. Which is why they were distributing the two corrupted PAL games I mentioned earlier.

The problem with pragmatism is it removes motivation for someone to go and dump and verify the game, so it ends up not getting done.

The second biggest issue is that they don't require proof and public disclosure of who dumped what. It's optional now, but for the most part, it's private. So if a ROM ends up being bad, we don't know what else that person verified to go back and audit their work.

Same. I always run SNES9x to see if it says good checksum on my roms.

That proves absolutely nothing, sorry. Those checksums were good on both the corrupted ROMs I mentioned. On one of them, someone used a tool to 'correct' the checksum. On the other, the bits flipped in such a way that the checksum was still the same.

A checksum inside of your file serves absolutely no purpose. Anyone can replace it with any checksum they want. A real checksum needs to be much stronger than 16-bits like the SNES uses, too. SHA256 has 1766847064778384329583297500742918515827483896875618958121606201292619776 times as many possible results as SNES checkums do (they have 65536 possibilities.) And said checksums need to be stored externally to your files, so that the files can't be patched with new checksums.

 

[byuu]

How's this for an update? This just arrived in my mailbox today.





Machines my ass. If the label fell off, it would be right next to the box without a label. Employees at the Jersey City location just stole a box containing $10,000 worth of games. This is felony theft.

I will be contacting both the press and a lawyer.

 

[NeOak]

How's this for an update? This just arrived in my mailbox today.





Machines my ass. If the label fell off, it would be right next to the box without a label. Employees at the Jersey City location just stole a box containing $10,000 worth of games. This is felony theft.

I will be contacting both the press and a lawyer.

Holy shit