Recent Loss of Data Incidents

@apollo-balthar He was talking about Grogmanay's long time to record completions when it first dropped, not the Gold and Glory.

You have to break eggs to make an omelette.

Thanks for the update.

I’d rather have 1000 Ancient coins instead of Gold and doubloons.

@wolfmanbush easily fixed by creating a ticket to Rare with video evidence of those achievements 😋

@captain-fixx

Oh for an ideal world.. Good job nothing serious was lost, my friend Karen was most put out though.. Shappens..

@captain-fixx

EDIT: One other explanation I can think of is too many concurrent connections attempting to write from the app/game servers to the DB side reward table and that is what times out from the app/game server side (though that is not what the explanation indicates to be true).

If that is the case, yeah then every timeout would be unrecoverable from a DB perspective but the app/game server should still have logs indicating the timeout and that should be recoverable.

The safeguard fix here is to ensure that all app/game servers write the rewards locally and retain for a period of XX days so if Rare sees the same DB issue again they can just replay them from the application once the DB side issue is resolved. Go so far as to put app/game side reward timeouts into their own log with the express intent of replay-ability of that data back into the engine.

You may be right there, whatever happened stuff didn't get chance to get put in the tables or it would have been recovered as you say. Recovering stuff from logfiles sometimes causes tears ime unfortunately.

Thanks for the technical update! I don't know a lot about this kind of stuff, but the seas certainly have felt laggier over the last few weeks.

It is getting harder and harder to maintain hope that you guys will ever get out of the hole and will be able to build the game into a bigger and better product. Between network limitations of events / ships, and hardware limitations of xbox one, I think you guys are going to need to make a tough call down the road about how to add content.

My suggestion is make new content in separate instances you load into. Other ships can follow you in and out, but it is a seperate entity from the main server. Then you might be able to design these instances differently so it doesn't break the "legacy game."

Still, appreciate the technical rundown and transparency. Love the game and just want to see it thrive.

@rcadden
You're leaving out so many factors that are outside of their control and then blaming them for it.

In order to stress test your programs you have to have people. They can't make people play at will in order to stress test their servers and programs. They've asked for it a couple of times in the past but they can only get so many people to participate.

What's happened in the past was on a smaller scale and they were able to straighten it out and get it sorted.

Next, you're thinking too small when you think they can just shut down servers and fix it. You have an entire world playing a game on servers in locations across the globe. Plus they don't own the server's. They're are Microsoft servers.

Plus there are 3 different platform types. Though the problem is similar it's not entirely the same thing because they have to locate and identify what's causing the problem this time and why. And when they say that they know of the problem it's purely reactionary didn't know about it ahead of time.

Are you going to award players for their losses? I hit every quest item in Grogmanay, but it never registered the gunpowder skeletons fully.

While it is sad to see the loss of data, especially in a game that relies entirely on goal completion, I understand that bugs can happen. However, I would like to see more compensation than a relatively small amount of gold. While I see the reasoning behind it, as you would not want to give out large quantities to people who did not experience data loss, I still feel that it does not do justice to those that worked on especially difficult commendations at that time. Ancient coins would be more appropriate for the compensation, as I feel that even 250 would be enough, as it costs you little to nothing, while essentially saving money for those that lost time.

On the point of the data loss itself, it is perfectly understandable as to the nature of the bug, since there is no way to test the effects of unprecedented amounts of players on the system without releasing it to the general public. I do, however, wish that the player base was told about it sooner, since that would have allowed for us to plan activities that are easier to redo, or not play at all for the short time it takes for a fix to come out.

It’s always great when you do 2 Athena voyages and the emissary after it to get a nice 20000 Gold compensation... i could care less give me one Athena level rank up instead.

@voodoo-ic0n they shut down the servers entirely, globally, every time an update is pushed, dude. It's a ~monthly occurrence. So they can do it.

Sure, it has some pretty major business impacts, but it's also more respectful to your playerbase - better than letting folks keep playing for things that are just going to disappear into the void.

I can not believe that people are defending this loss of data.
There are no quick sessions in this game. You sink a lot of time into it to reach certain goals and for those actions not being counted you have effectively had your time wasted.

For example, Running Athena Emissary, in an attempt to chip away at the the rep grind to A20 and get the one item I have left to collect 50 of, the Ancient Villainous Skull. After doing an Athena run and getting 0, we drop another we get 1. (The fact they are so ridiculously rare is another slap in the face of Pirate Legends and warrants it’s own looking into) but, Lo and behold, the item doesn’t get counted and Athena rep wasn’t tallied. So we got gold and doubloons, instead?

We don’t want the gold and doubloons, we want what we were working towards. On top of it, we get gaslit by moderators and fanboys, pretending like it an not an issue, telling us to just “enjoy the experience”. We are literally playing the game Rare put out, working on the goals they set forth. We spend time and some spend extra money in the emporium as well. This better get fixed by the time the Seasons start because I can see a lot of people not participating at all if they aren’t getting the only rewards this game offers.

Was scratching my head why my stuff wouldn't record.... seems queuing systems need incoming data trail so transactions can be re-created in emergency , odd no backups of the trail in this "cloud" age.

Was last weekends last minute double XP event a test to try and work out what had gone wrong the first time?
Feels abit like my lost Athena Emmisary run last Friday night was purely so you could see where your problems were coming from.

I'm a sucker and played again the next day to make sure I got my Athena stuff, but honestly I'd love to see the basics sorted and made sure they're water tight in the game before Seasons start. Absolutely appreciate people are still working from home which makes things a million times more difficult, but if that is the case instead of pushing ahead with plans that overcomplicate, I just want the game to work and log what I do and not feel like I've wasted any time on the Sea ☺️

@voodoo-ic0n Thank you for speaking some reason.

@whyo Listen dude, I get the issue, I lost progress as well. It susks for sure. But this post literally states that a system was having issues and they are doing everything they can to fix it. Its their job to fix it and they know what we lost. Unfortunately Theres no real efficient way to somehow give back exactly what players lost, it's just unrealistic.

@calicorsaircat I completely agree with you. They are going to have to make a decision about hardware restrictions. Might be the only way for this game to reach its truest potential, and it has a lot of potential. On the other hand, I can understand if they would want to keep the game accessible to more audiences, in a way it's more consumer friendly.

@rcadden
But that's because you're just assuming that it's the server itself and not a piece of code that they can fix on the Fly.

Ever heard of a "Hot Fix"? It's when they fix code as the servers are still live. Ultimately that's what they're doing is running hot fixes and waiting for the scheduled maintenance window where they can shut down the servers and do other fixes as necessary.

Shutting down the servers is not always the solution.

@jack-henrei listen guy, it was irresponsible for them to run it a second time, without even letting us know. People were in the dark and working towards goals unaware that it wasn’t being counted. Stop making excuses for their poor judgement. If people would have known there was a possibility it wouldn’t count we could have gone and done something else with our time. I play casually, some people I’ve met in game were grinding for like 20 hrs.
Oops, sorry guys we lost all your data. Oops, we did it again. Here’s some sheckles for your lost time.

@whyo My point is this, you're pretty much coming on the forum soley to continue spreading your distaste for what happened, underneath a post explaining and apologizing for what happened. See what I am getting at? How would they have known the system would fail? And, if you read the post, they needed to run the system a second time to figure out the problems and fix it. A warning in the future would be a welcomed one, but a majority of your points are based around being annoyed you lost stuff. Everyone is annoyed they lost stuff bro.

@jack-henrei you’re literally proving my points. There are people who will refuse to tolerate any criticism about this game, so they defend Rare no matter what. They didn’t notify the community that there was a possible loss of the data the second time they implemented the system. The first time it happened it’s an accident they ,didn’t know. The second time it happened, that’s a risk they took and they wasted people’s time.

Forums exist for feedback, good or bad. Deal with it. You can always scroll past the posts that get your jimmies rustled.

To the customers that don't get the tech behind these games:

They care A LOT about what happened. That is why Rob wrote this. Please realize that if they DID NOT CARE Rob would not have written a public note explaining the situation. Many, many companies wouldn't have gone this far (I know, I have been in the tech industry for quite a while).

Behind the scenes:
They are doing their absolute best to capture all the lessons learned, execute a roadmap to get resources to fix them and make sure it doesn't happen again. The best technology teams used continual improvement processes to try to exploit failures to get better. By doing this every time, it leads to excellence.

My advice? Just ask them to make it personal and engineer a better system because we love the game and we trust they have a high performing, devoted team that cares...

See you on the SEAS!

CutterBCB

You apparently need to take some real, solid looks at the way you design and code this game. It's riddled with pathetic bugs and horrible gameplay mechanic failures, and for how long? Whoever is in charge of these issues should think about finding another job. Let someone who actually has a sense of direction and clearly defined standards take over so they can impose a level of quality among the people working under them, because this is just embarrassing.

@whyo In all sincerity I respect your post, I only reply to posts I find bring up important discussions, my only intention was to bring a bit of reason and cool down a majority of angry players, because it is our job as the playerbase to call bs when it is needed. I apologize as I came off too strong on the defense. I felt some of the anger towards rare however was not actually working towards the wider issue. Which is communication. The root of what seems to be most of this games biggest problems somehow. I feel as if they used to really SPEAK with us in the early days, every month we hear less and less. What I really fell in love with the game for originally was their honesty and passion for the community, I hope that passion is not sizzling out.

@jack-henrei it’s all good no worries. I was a little aggressive in my reply, so I apologize as well. I agree with what your saying. What I’ve learned playing this game and being in the forums since launch, is there are key issues with the game that Rare simply refuses to address. That’s why it’s hard for me to give them the benefit of the doubt sometimes.

This is still the number one game I play. I’ve never really taken a break from it. It’s just ridiculous that these issues aren’t being announced to the community. I had to take a screen shot of my money tonight and with all my loot on board because I was getting zero money and zero emissary value. I’ll check tomorrow when I log on. Hell, it’s been three years and they still haven’t fixed the hit registration 🙄. That being said, the new content is fun and for the first time in a long time the game made me feel like I did when it was brand new. I like the season so far and what they are doing towards events but I’d rather them take a break from content creation for a little and just fix all of the things that don’t work well. Fair winds and following seas my friend.