The game was crashing. Not a lot, but about once a day the game would randomly just explode. No particular rhyme or reason to the stack trace or what was going on at the time. Just, boom!
We obviously can't ship that way. It was getting kinda late in the development cycle, enough that this made everyone nervous, and I was tasked with tracking the thing down. They said we hope you find it fast, but fast or slow, you have to find it because we don't really have a choice.
I spent about a week trying to see if it was related to any particular thing. I started disabling subsystems to track it down. No audio, wireframe graphics, simple test level, remove all equipment or controllers we don't need. Nope. At this stage you start having those tense conversations like "Maybe we don't NEED level 6.3" when fiddly bugs start to show up, but it was irrelevant, because nothing we could remove would solve it. I made some level of attempt to run back in the source control to see if I could binary-search to where it first showed up, but the game crashing isn't exactly unheard of and it was fiddly to even reproduce the thing in the first place. That was what made it so insanely slow to track down. No luck from any of this.
After several days, I understood that I wasn't in for a typical debugging session. What I decided on was to embark on making a build of the game that was byte-for-byte identical from run to run. No audio, no control input, constant RNG seed, track down anything that might make it deviate or anything time- or IO-dependent. That took about another week, but at the end, I had my build. Every byte of memory was the exact same from run to run; every address, every value. (This was a console game so this task was easier than it otherwise would have been.)
So we're two weeks in now. By dumb luck, I managed to replicate the bug almost immediately once I had the repeatable build. The morbidly simple level and setup that reproduced it quickly was: Start the repeatable build with a script running that would spawn the player in a bare room with a floating gun. The gun fires, killing the player. The player drops to the ground, and then the game crashes. Every time.
I got you now, you son of a bitch. Happy that the thing was trapped now, unable to flee into its stochastic wilderness, I began to live in an endless world of execution-style slayings.
What followed was actually the longest part. As I said, the crash had no real pattern, but I now had the ability to go backwards in time. I could control everything. I took any wrong-looking values from the memory at the time of the crash, and started setting watchpoints on their memory addresses. How had it gotten this way? What touched this piece of memory? The process of single-stepping through, as a single piece of memory was allocated, used, freed, and then reallocated, trying to understand it all and find the wrong bit, was intense.
My memory of the following couple of weeks is honestly a little fuzzy. I know I was able to track down, by single-stepping through every single place the offending memory value was touched, to definitively work out what value it was that had had an impossible value, how it had caused the crash, and every single place that value had been touched, until I arrived at the offending code. The problem was... the bug wasn't in the code I was looking at. It had had some weird impossible value in some of its data that had caused it to set up something else wrong.
Oh. Oh no. Do I have to do it again? By now I was three or four weeks in.
I did do it again. Like I say I don't remember clearly, but I remember that it took me about 6 weeks in total and I filled up the majority of one little notebook that I kept next to the computer to write notes in. Little hex addresses, stack traces, clues that I might need to flip back to later. No matter. After tracing back all the breakage, I finally arrived at the code that was actually causing it. Once I actually saw what was happening, it was easy to work out.
At some point, someone had added a cache for some expensive-to-compute data for actors in the world. The cache had a mapping of actors to cached-value matrices, and each actor had a single value that was a pointer to its cached value, or NULL if it had none. I think they were too big to store indefinitely? There was a little LRU cache. But, if an actor was destroyed, it wasn't removed from the cache. So its cached data would sit there in the LRU cache, like a little time bomb, until the thing expired, and then the last part of its removal from the cache would be... unsetting the pointer to cached data in the now-dead actor back to NULL. The actor's memory had already been freed, and so one int, somewhere at some random location in memory, would get set to 0. And once in a while, it would happen to be at a location where that would cause serious problems (usually leading quickly to a crash).
Hm... not sure it would have. Something simple like overwriting freed memory with an identifiable pattern wouldn't have helped, because by the time the crash happened, the actor's memory had been reallocated and was being used for something else.
This was on the Wii, which had some pretty bare-bones development tools, so it wouldn't have been real easy to try to use something like valgrind to do more sophisticated memory debugging or anything. It was basically single-threaded, too, BTW (I think audio and input were based on interrupt handlers), and I made it more so with the repeatable build.
Yeah. I was proud of myself. 😃 IDK how easy it is now with all these friendly toolkits, but back at that time and definitely for console development, there wasn't a lot of way around really knowing your shit. Even having a degree, it definitely helped, but even that wasn't really a full preparation for some of the difficult stuff I ran into. I wish CS prepared people better for real-world engineering; in particular I wish reading big codebases was more of the curriculum. I actually came out of school still with fairly bad working-with-production-code abilities.
Separate related story, I was disagreeing with my boss about whether to hire some guy. My boss said, but his code is really bad. I said, yeah but he's right out of school so it's to be expected. When I first graduated my code was really bad. My boss got sort of distant for a second while he was remembering, and then his face came alert again and he said, all animated, "Yeah! It was. It was terrible!" I said yeah, I know, then I learned what I was doing and it was better. It's just sort of how it works.
I'm excited to see more once it becomes available. Have already added to wishlist. A couple of thoughts, similar to what some others have voiced:
Animations are not everything or enough to make a good game. However, attention to detail in animations is a big plus in my book, regardless of graphical quality. It is something that I tend to notice and appreciate.
Mechanics are fundamental to any game, physical or digital. There's two diametrically opposed directions that I've seen that result in enjoyable games:
Mechanics that are optimized for enjoyment rather than strict realism. People play games for entertainment. If a mechanic is extremely frustrating, it will likely cause people to lose interest even if it is extremely realistic. In this approach, such mechanics are tuned or dropped to ensure that the player experience is as good as it can be.
Mechanics that are optimized for realism to an excessive degree, with enjoyable gameplay taking a backseat. This is likely to result in a smaller, cult following as many will get frustrated and move on. A good example of this is Dwarf Fortress with its unofficial slogan of "Losing is Fun".
I would strongly suggest leaning towards the former as the latter is a really hard target to hit and the cult following for a game that does hit #2 perfectly may also be delayed until long after release.
Graphical quality can be very overrated. An enjoyable game with terrible graphics and animation may be very replayable while one with stunning visuals and terrible gameplay may be a flash in the plan that is quickly forgotten. I highly recommend ensuring that graphics are well-optimized so as to allow stutter-free gameplay on even low spec machines, if settings are sufficiently reduced.
Keep up the good work and I look forward to seeing your future progress.
Thank you so much! Yeahabout the realism there will be difficulty levels option like walker,survivalist,expert and etc. About the optimization yeah i am always pay attention to that while developing too. Thanks for the wishlist! 👍
Hey! So, I'll go into more detail once I have a strong enough connection to watch the videos smoothly. For now some general advice that is often missed by new game devs:
Make your game fun. Not just fun to you; fun to others too. Even if that means taking a step back and looking at a mechanic you personally enjoy from a very objective perspective.
Make sure the game runs well. Limit the jittering and framerate hiccups. Give flexibility to controls. Let people change them. Provide all the options.
Drip feed mechanics into the gameplay in a natural way BUT make sure everything is technically unlocked immediately. This opens up more replayability without the opening slog some games force people to sit through.
Finally, ask for help and take criticism seriously. None of us are shitting on something you have put tremendous work into. We want the experience to be awesome. As much for you as for us.
Bonus: If you can, try and release your game on Steam between other releases in the same, or in adjacent, genres. Also consider reaching out to streamers that have shown they are willing to give factual and fair critiques, reviews, and chances for a game.
I have been considering creating a game for some time and these are the rules I would follow. Fun factor and playability absolutely stomp on everything else. A fun, playable, easy to get into experience will sell itself. From there it's just a matter of holding onto the hype long enough to make all that effort so very much worth it.
Hi, i read it all of your thoughts. İ really appreciate it that you shared your valuable time while writing this. Yeah, i totally agree with you, you are 100% right. İ will try my best to figure it out. Have a nice day.
I'm certain you'll do great! Creating a video game is a tremendous undertaking, and you are risking a lot more than your time and energy by making this. All of us want that effort and those sleepless nights to work out for you. :)
For what it’s worth, creating a game that is fun to yourself will almost certainly resonate with other gamers that are like you. It can make a game bad if you focus on the general population instead of your own personal passion project.
This of course assumes you have good taste and high standards of polish.
In my worthless opinion, if the game is functioning as intended AND you get those sweet advertisements out there. It will probably sell pretty alright. Survivals are big right now.
Hi, Yeah as a survival games lover i am trying to make a game that you never saw on other survival games. For example, did you see any survival game with the cooking meet animations like that? İt is kot final result. İ will polish it. Thanks
Dont count on animations selling your game, my guy. It helps but its no bandaid for addictive gameplay. There could be no animation at all and if its fun I'll still play it. Make sure the mechanics bring about that feeling of completing tasks and advancing and it'll do fine.
I don't mean to be passive-aggressive, animations are really hard (especially to produce on game-wide scale alone), but I love those in-sync doggos, choreographed tails like they are about to start a (good-)boy band.
They look like they're about to start snapping their fingers menacingly at 0:46.
While we're talking about it, at 0:16 on the first video it kinda looks like the PC just sticks their hand in between the wolf's jaws, before punching it with their other hand. Got a chuckle from me.
I have not heard "snapping their fingers menacingly" since I heard a description of West Side Story. Pleased to come across it again, this is going to make me actually check out the trailer now
I actually surprised that when I went looking for a clip to share along with that comment, I couldn't actually find any clip of them walking directly at the camera doing it like it happens in my head. (I've never actually watched it either on stage or film, but I guess I know it through cultural osmosis.)
Given the obvious similarities to The Long Dark (even your title nods to the influence), what differs your game from your inspiration? What did you want to expand upon or change?
Not only do the games need to be opensource, but their toolchains too. It would also be great if everything were stored in content-addressable storage. Then it wouldn't be necessary to track down stuff by follow URL paths to dead domains.
I hope that opensource game engines become more popular. It might just be a matter of time. Blender is now very popular and that's opensource.
Game Development
Hot
This magazine is from a federated server and may be incomplete. Browse more on the original instance.