As a software developer, here's an inside secret for you guys: QA is really hard.
I totally disagree. They only changed 1 thing on the AH. 1 thing. That thing clearly failed it's smoke test.
The person who wrote the smoke test and didn't account for this scenario should be fired.
Lack of transaction amount verification in the app should get the lead developer/designer fired as well.
Since this is a long string of fumbled balls, I'd be shocked to not see someone in Senior Management fall on their sword. Guessing Jay Wilson, given how Corporate already knows hes an unliked persona whose departure would cool a lot of people down who are going to quit over this.
As a software developer, here's an inside secret for you guys: QA is really hard.
I totally disagree. They only changed 1 thing on the AH. 1 thing. That thing clearly failed it's smoke test.
The person who wrote the smoke test and didn't account for this scenario should be fired.
Lack of transaction amount verification in the app should get the lead developer/designer fired as well.
Since this is a long string of fumbled balls, I'd be shocked to not see someone in Senior Management fall on their sword. Guessing Jay Wilson, given how Corporate already knows hes an unliked persona whose departure would cool a lot of people down who are going to quit over this.
Not sure what to say other than this seems like the worst mistake I can remember in online gaming, like...ever.
So we're not counting SimCity failing to work for long periods of time and when it did it had half hour queues or more (assuming you didn't get booted after the wait). A game which had at lest one main feature removed becuase their online system couldn't even handle it.
Or maybe even Neverwinter who's crashed several times this week and has a website so full of bugs that not only does it not complete crafating but fails to reward players their in game currencies that are vital to its economy (and will probably not be refunded to players)? Not to mention that outcry people have about their pricing practices.
Or what about the recent Farcry 3 Blood Dragon leak that happened on Ubisoft's own website where it was found weeks before launch and may have crippled it's online sales?
All this compared to gold in diablo.. which lost its gold value.. got caught and stopped fast and is taking steps to repair the damage done? Gold losing its value is the WORST multiplayer issue ever? I do not comprehend this logic. Granted that last one wasn't a "multiplayer game" issue but it does show that far worse mistakes have been made than a economic crash in a game where the devs want players to move away from the AH anyway. Just sayin'
As a software developer, here's an inside secret for you guys: QA is really hard.
I totally disagree. They only changed 1 thing on the AH. 1 thing. That thing clearly failed it's smoke test.
The person who wrote the smoke test and didn't account for this scenario should be fired.
Lack of transaction amount verification in the app should get the lead developer/designer fired as well.
Since this is a long string of fumbled balls, I'd be shocked to not see someone in Senior Management fall on their sword. Guessing Jay Wilson, given how Corporate already knows hes an unliked persona whose departure would cool a lot of people down who are going to quit over this.
They won't rollback.
Not sure if you were aware but Jay is on other projects now and wouldn't have been the one working on this part of the game. Honestly I can agree that it was an easy thing to miss, but my guess is because it was added last second (this didn't show up on any ptr tests that I'm aware of) that it was just missed by pure accident.
Blizzard just keeps One upping themselves. Just when I thought they possibly couldnt ruin this game any more they do. I seriously dont see anyone staying around once PoE comes off beta and is live. Having played D3 since launch and having 3 lvl 70+ characters in PoE I honestly cant think of a single thing Diablo 3 has done thats even close to being better than PoE. Also having a company that actually cares and listens to its players is just icing on the cake. But I will still log in and play D3 with some false hopes that one day you wont need the AH or real money to actually get gear and it will come from mobs rather than the AH.
So we're not counting SimCity failing to work for long periods of time and when it did it had half hour queues or more (assuming you didn't get booted after the wait). A game which had at lest one main feature removed becuase their online system couldn't even handle it.
That game never really got off the ground. This was a patch that effectively nuked in 2 hours, an economy that had been built over the last full year of a live game.
Or maybe even Neverwinter who's crashed several times this week and has a website so full of bugs that not only does it not complete crafating but fails to reward players their in game currencies that are vital to its economy (and will probably not be refunded to players)? Not to mention that outcry people have about their pricing practices.
It's in beta? This is a live release? How can you compare the two?
Or what about the recent Farcry 3 Blood Dragon leak that happened on Ubisoft's own website where it was found weeks before launch and may have crippled it's online sales?
All this compared to gold in diablo.. which lost its gold value.. got caught and stopped fast and is taking steps to repair the damage done? Gold losing its value is the WORST multiplayer issue ever? I do not comprehend this logic. Granted that last one wasn't a "multiplayer game" issue but it does show that far worse mistakes have been made than a economic crash in a game where the devs want players to move away from the AH anyway. Just sayin'
Unintended consequence, live game, small part of the patch, completely annihilated the economy, and had real world dollar impact. If you can name me another that has those qualities, let me know.
Not sure if you were aware but Jay is on other projects now and wouldn't have been the one working on this part of the game. Honestly I can agree that it was an easy thing to miss, but my guess is because it was added last second (this didn't show up on any ptr tests that I'm aware of) that it was just missed by pure accident.
I was pretty much just speculating on them picking a fall guy. If I was management, there are few other "names" in the D3 staff that would get the company more mileage by canning. At least, from a PR standpoint.
As a software developer, here's an inside secret for you guys: QA is really hard.
+1 to this too. QA is hard, I have been involved in software dev and testing for over 20 years. But really this type of thing is easy to catch... there useless and cheap for a product this large. All you need is to do is a few basic things;
1) snapshot database (base)
2) In 1.07, buy/sell, cancel and perform a bunch of transactions etc on AH.
3) snapshot 1.07 data base (DB post transactions 1.07)
4) revert to base snapshot
5) Patch 1.08
5) In 1.08 repeat same transactions as 1.07
6) snapshot 1.08 data base (DB post transactions 1.08)
7) Compare database and make sure any differences are as expected
If they did this half the problems they have would not occur.
This trend of fckups will continue so just laugh it off and avoid investing anything big into it and you won't get disappointed. You're just a delusional fool if you think anything else.
Or... it doesn't matter.
D2s "economy" was obliterated by duping within a month and the D2 team was completely unable to do anything about it for the vast majority of the game. No one gave a shit. We played the game and had fun instead of acting like little drama queens who have to be outraged at EVERYTHING.
Actions are already being taken. Exploiters are banned in waves and transactions are being reversed.
trillions of dup gold is already in the playerbase just from gold-to-gold transactions alone. what this works out to is everything costing 10x as much as it used to but non-cheaters having 1/10th the buying power they had yesterday.
This is probably something that dev should have asked QA to test. I'm not necessarily saying it's QA's fault, but it is definitely somebody's fault. Don't they have test case reviews where people are supposed to point out things which need to be tested? If so, why wasn't one of them "sell a large amount of gold and then cancel it to make sure the refund works properly"? Somebody clearly dropped the ball here.
The Devs wouldn't have considered that to be a problem. They didn't adjust anything to do with the Method that controls cancelling auctions. That Function should have remained as it was prior to any other changes, all that would change is what number got passed through it.
I'm not saying it wouldn't have been a good thing to test, I'm just saying these things can get missed when you've been staring at the same code and program for a long time. It's the same way that typos and such keep happening in books, news, and the like. There was probably some quick testing to make sure the right amount was taken out and given to the buyer, but the part where you cancel and make sure it goes back to inventory probably got missed. I've seen it happen to even very detailed people, becuase certain things are assumed to work.
I guess my point is that programming errors happen ALL THE TIME and just becuase it's Blizzard people feel the need to elevate it to inane perportions. They're typically quick to fix critical errors even if smaller ones take longer (due to their QA process).
+1 and quoted for truth. hehe. My programming never got to pro levels, but I saw first hand many pitfalls that could catch people and had a really good professor that was highly experienced in the trade. I learned a lot from that man.. May he rest in peace.
That's not how you do QC. You test the things that have changed in that patch. The RMAH had no changes. The biggest priority is always things that make the game unplayable or break your account. You can't have an endless list of things that need checking during each patch because small patch iterations go out a few times a day during development, and the "final patch" doesn't stay on the internal severs for more then a day or few, because it's delayed content if it's ready.
With all that said the change could've been accidental and not present on the internal severs at all. Many things happen when launching a game with a super complex engine like Diablo's.
To be clear guys, I have a M.S. in Computer Science and I work at a Fortune 500 company in engineering / internal and external tech support for hardware and software we develop. Part of my job is to attend test case reviews to make sure stuff like this doesn't get missed, so I do have some clue what I'm talking about here. Bugs go live all the time, but when they are as devastating as this one, there are (and should be) inquiries as to how it happened. I'm not calling for firings, but there is obviously a flaw in their testing process. Every aspect of the AH should be tested and retested every time even a minor change is made to it because of the amount of money involved and the catastrophe which will occur if any abusable bugs are discovered. It's not exactly rocket science to figure out that when some part of the mechanism for selling gold for real money changes (and it did in this case), you need to test said mechanism. I assume they at least tested basic functionality, but they clearly did not test it comprehensively enough and should reexamine their process. Are you guys arguing that because testing is hard, they shouldn't try to find out why this happened and learn from it, because that's what I'm hearing.
Rollback Post to RevisionRollBack
...and if you disagree with me, you're probably <insert random ad hominem attack here>.
This was a massive failure and no talk can reverse that. Hopefully Blizzard does the right thing and rolls back every account, but whatever they do, however they minimize the effect, the fact stays: even after one year they can't do a patch without major bugs, however small it is.
> 10:15 p.m. PDT: We believe we’ve found a fix for the gold duplication bug and will be deploying it shortly to all regions. In order to implement this fix, we'll be bringing Diablo III into maintenance. We anticipate this maintenance will last for approximately 1 hour, lasting until approximately 11:30 p.m. PDT.
Are you guys arguing that because testing is hard, they shouldn't try to find out why this happened and learn from it, because that's what I'm hearing.
I think they're arguing that finding these sort of issues isn't as simple as walking down the patch notes and running a few tests for each bullet point... even taking into account that the patch notes we see are by no means the entire changelist for the corresponding release.
On the other hand, the AH part of the code in particular should be smothered in unit tests, acceptance tests, million-monkey tests, blood tests and urine tests.
So yeah... QA is hard, but finding these sort of issues is a solved problem, and TBH (having played WoW for many, many years) it seems to me that right across the board, Blizzard's QA program doesn't have nearly as much (automated) regression testing as it should, and leans a little too heavily on PTRs (which should be for design iteration, not bug-spotting).
I only read OP and a few posts here and there, so if someone addressed this. I apologize for the redudancy.
I agree that it is partially Blizzard's fault. But by agreeing to the terms of service you acknowlege the account is RENTED. You do not own any of the characters, items, or anything on it. You agreed not to abuse any glitch or dupe and to abide by the rules. In turn, it's ALSO your fault. Blame is on both ends. End of story. Anyone who knowingly duped gold should definitely have their account banned.
This is probably something that dev should have asked QA to test. I'm not necessarily saying it's QA's fault, but it is definitely somebody's fault. Don't they have test case reviews where people are supposed to point out things which need to be tested? If so, why wasn't one of them "sell a large amount of gold and then cancel it to make sure the refund works properly"? Somebody clearly dropped the ball here.
@Mamba: We don't know. I think Blizzard will find some way to resolve this. I just can't say for sure what it will be yet. It appears that they are trying to avoid rolling back, and it sounds like at least some paypal transactions are being rolled back, but we have no real way to know right now.
That's not how you do QC. You test the things that have changed in that patch. The RMAH had no changes.
Didn't they change the gold limit for auctions? I mean that's the whole reason this dupe was possible. They fiddled with the AH and created some overflow error. They didn't bother testing and here we are.
I am truely sad honestly....I hope blizzard gets themselfs together again soon!
I WANT OLD BLIZZARD BACK !
"The game is done when its done"
For being speechless, there was quite a lot you had to say. Although you didn't really say anything meaningful. My favorite part is "gold duping... in Diablo... how can this even happen..." and "I want the old Blizzard back". => Maybe you should go and install the older Diablo of the old Blizzard that you want "back". Hint: Gold duping in Diablo 1 was the easiest thing ever. Sure you want that back?
Man, there are so many things so wrong about this video, please just stop. I'll stop here myself because everything I would have to say would get me an infraction.
This is probably something that dev should have asked QA to test. I'm not necessarily saying it's QA's fault, but it is definitely somebody's fault. Don't they have test case reviews where people are supposed to point out things which need to be tested? If so, why wasn't one of them "sell a large amount of gold and then cancel it to make sure the refund works properly"? Somebody clearly dropped the ball here.
@Mamba: We don't know. I think Blizzard will find some way to resolve this. I just can't say for sure what it will be yet. It appears that they are trying to avoid rolling back, and it sounds like at least some paypal transactions are being rolled back, but we have no real way to know right now.
That's not how you do QC. You test the things that have changed in that patch. The RMAH had no changes.
Didn't they change the gold limit for auctions? I mean that's the whole reason this dupe was possible. They fiddled with the AH and created some overflow error. They didn't bother testing and here we are.
I just don't like when we 100% blame Blizz, when it's the greedy exploiters whom are to blame for taking advantage. I hope they all get perma-banned, chars wiped....fucking gone. Whatever it is about the nature of people like this, they never seem to grasp the concept of consequence.
I say...make an example out of them and destroy their accounts with the same disregard that these players attempt to destroy our game.
Not sure what to say other than this seems like the worst mistake I can remember in online gaming, like...ever.
I totally disagree. They only changed 1 thing on the AH. 1 thing. That thing clearly failed it's smoke test.
The person who wrote the smoke test and didn't account for this scenario should be fired.
Lack of transaction amount verification in the app should get the lead developer/designer fired as well.
Since this is a long string of fumbled balls, I'd be shocked to not see someone in Senior Management fall on their sword. Guessing Jay Wilson, given how Corporate already knows hes an unliked persona whose departure would cool a lot of people down who are going to quit over this.
They won't rollback.
Prepare to get shocked.
Ha. Bagstone.
So we're not counting SimCity failing to work for long periods of time and when it did it had half hour queues or more (assuming you didn't get booted after the wait). A game which had at lest one main feature removed becuase their online system couldn't even handle it.
Or maybe even Neverwinter who's crashed several times this week and has a website so full of bugs that not only does it not complete crafating but fails to reward players their in game currencies that are vital to its economy (and will probably not be refunded to players)? Not to mention that outcry people have about their pricing practices.
Or what about the recent Farcry 3 Blood Dragon leak that happened on Ubisoft's own website where it was found weeks before launch and may have crippled it's online sales?
All this compared to gold in diablo.. which lost its gold value.. got caught and stopped fast and is taking steps to repair the damage done? Gold losing its value is the WORST multiplayer issue ever? I do not comprehend this logic. Granted that last one wasn't a "multiplayer game" issue but it does show that far worse mistakes have been made than a economic crash in a game where the devs want players to move away from the AH anyway. Just sayin'
Not sure if you were aware but Jay is on other projects now and wouldn't have been the one working on this part of the game. Honestly I can agree that it was an easy thing to miss, but my guess is because it was added last second (this didn't show up on any ptr tests that I'm aware of) that it was just missed by pure accident.
That game never really got off the ground. This was a patch that effectively nuked in 2 hours, an economy that had been built over the last full year of a live game.
It's in beta? This is a live release? How can you compare the two?
Again that game wasn't live yet.
Unintended consequence, live game, small part of the patch, completely annihilated the economy, and had real world dollar impact. If you can name me another that has those qualities, let me know.
I was pretty much just speculating on them picking a fall guy. If I was management, there are few other "names" in the D3 staff that would get the company more mileage by canning. At least, from a PR standpoint.
1) snapshot database (base)
2) In 1.07, buy/sell, cancel and perform a bunch of transactions etc on AH.
3) snapshot 1.07 data base (DB post transactions 1.07)
4) revert to base snapshot
5) Patch 1.08
5) In 1.08 repeat same transactions as 1.07
6) snapshot 1.08 data base (DB post transactions 1.08)
7) Compare database and make sure any differences are as expected
If they did this half the problems they have would not occur.
I'll take that bet, name the stakes!
Or... it doesn't matter.
D2s "economy" was obliterated by duping within a month and the D2 team was completely unable to do anything about it for the vast majority of the game. No one gave a shit. We played the game and had fun instead of acting like little drama queens who have to be outraged at EVERYTHING.
trillions of dup gold is already in the playerbase just from gold-to-gold transactions alone. what this works out to is everything costing 10x as much as it used to but non-cheaters having 1/10th the buying power they had yesterday.
Banning is not enough to fix this.
To be clear guys, I have a M.S. in Computer Science and I work at a Fortune 500 company in engineering / internal and external tech support for hardware and software we develop. Part of my job is to attend test case reviews to make sure stuff like this doesn't get missed, so I do have some clue what I'm talking about here. Bugs go live all the time, but when they are as devastating as this one, there are (and should be) inquiries as to how it happened. I'm not calling for firings, but there is obviously a flaw in their testing process. Every aspect of the AH should be tested and retested every time even a minor change is made to it because of the amount of money involved and the catastrophe which will occur if any abusable bugs are discovered. It's not exactly rocket science to figure out that when some part of the mechanism for selling gold for real money changes (and it did in this case), you need to test said mechanism. I assume they at least tested basic functionality, but they clearly did not test it comprehensively enough and should reexamine their process. Are you guys arguing that because testing is hard, they shouldn't try to find out why this happened and learn from it, because that's what I'm hearing.
> 10:15 p.m. PDT: We believe we’ve found a fix for the gold duplication bug and will be deploying it shortly to all regions. In order to implement this fix, we'll be bringing Diablo III into maintenance. We anticipate this maintenance will last for approximately 1 hour, lasting until approximately 11:30 p.m. PDT.
I think they're arguing that finding these sort of issues isn't as simple as walking down the patch notes and running a few tests for each bullet point... even taking into account that the patch notes we see are by no means the entire changelist for the corresponding release.
On the other hand, the AH part of the code in particular should be smothered in unit tests, acceptance tests, million-monkey tests, blood tests and urine tests.
So yeah... QA is hard, but finding these sort of issues is a solved problem, and TBH (having played WoW for many, many years) it seems to me that right across the board, Blizzard's QA program doesn't have nearly as much (automated) regression testing as it should, and leans a little too heavily on PTRs (which should be for design iteration, not bug-spotting).
I think this is not the first time the US players function as lab rats for us EU players. Kinda sad.
I agree that it is partially Blizzard's fault. But by agreeing to the terms of service you acknowlege the account is RENTED. You do not own any of the characters, items, or anything on it. You agreed not to abuse any glitch or dupe and to abide by the rules. In turn, it's ALSO your fault. Blame is on both ends. End of story. Anyone who knowingly duped gold should definitely have their account banned.
Didn't they change the gold limit for auctions? I mean that's the whole reason this dupe was possible. They fiddled with the AH and created some overflow error. They didn't bother testing and here we are.
For being speechless, there was quite a lot you had to say. Although you didn't really say anything meaningful. My favorite part is "gold duping... in Diablo... how can this even happen..." and "I want the old Blizzard back". => Maybe you should go and install the older Diablo of the old Blizzard that you want "back". Hint: Gold duping in Diablo 1 was the easiest thing ever. Sure you want that back?
Man, there are so many things so wrong about this video, please just stop. I'll stop here myself because everything I would have to say would get me an infraction.
I just don't like when we 100% blame Blizz, when it's the greedy exploiters whom are to blame for taking advantage. I hope they all get perma-banned, chars wiped....fucking gone. Whatever it is about the nature of people like this, they never seem to grasp the concept of consequence.
I say...make an example out of them and destroy their accounts with the same disregard that these players attempt to destroy our game.
BurningRope#1322 (US~HC) Request an invite to the official (NA) <dfans> Clan