The game was crashing. Not a lot, but about once a day the game would randomly just explode. No particular rhyme or reason to the stack trace or what was going on at the time. Just, boom!
We obviously can't ship that way. It was getting kinda late in the development cycle, enough that this made everyone nervous, and I was tasked with tracking the thing down. They said we hope you find it fast, but fast or slow, you have to find it because we don't really have a choice.
I spent about a week trying to see if it was related to any particular thing. I started disabling subsystems to track it down. No audio, wireframe graphics, simple test level, remove all equipment or controllers we don't need. Nope. At this stage you start having those tense conversations like "Maybe we don't NEED level 6.3" when fiddly bugs start to show up, but it was irrelevant, because nothing we could remove would solve it. I made some level of attempt to run back in the source control to see if I could binary-search to where it first showed up, but the game crashing isn't exactly unheard of and it was fiddly to even reproduce the thing in the first place. That was what made it so insanely slow to track down. No luck from any of this.
After several days, I understood that I wasn't in for a typical debugging session. What I decided on was to embark on making a build of the game that was byte-for-byte identical from run to run. No audio, no control input, constant RNG seed, track down anything that might make it deviate or anything time- or IO-dependent. That took about another week, but at the end, I had my build. Every byte of memory was the exact same from run to run; every address, every value. (This was a console game so this task was easier than it otherwise would have been.)
So we're two weeks in now. By dumb luck, I managed to replicate the bug almost immediately once I had the repeatable build. The morbidly simple level and setup that reproduced it quickly was: Start the repeatable build with a script running that would spawn the player in a bare room with a floating gun. The gun fires, killing the player. The player drops to the ground, and then the game crashes. Every time.
I got you now, you son of a bitch. Happy that the thing was trapped now, unable to flee into its stochastic wilderness, I began to live in an endless world of execution-style slayings.
What followed was actually the longest part. As I said, the crash had no real pattern, but I now had the ability to go backwards in time. I could control everything. I took any wrong-looking values from the memory at the time of the crash, and started setting watchpoints on their memory addresses. How had it gotten this way? What touched this piece of memory? The process of single-stepping through, as a single piece of memory was allocated, used, freed, and then reallocated, trying to understand it all and find the wrong bit, was intense.
My memory of the following couple of weeks is honestly a little fuzzy. I know I was able to track down, by single-stepping through every single place the offending memory value was touched, to definitively work out what value it was that had had an impossible value, how it had caused the crash, and every single place that value had been touched, until I arrived at the offending code. The problem was... the bug wasn't in the code I was looking at. It had had some weird impossible value in some of its data that had caused it to set up something else wrong.
Oh. Oh no. Do I have to do it again? By now I was three or four weeks in.
I did do it again. Like I say I don't remember clearly, but I remember that it took me about 6 weeks in total and I filled up the majority of one little notebook that I kept next to the computer to write notes in. Little hex addresses, stack traces, clues that I might need to flip back to later. No matter. After tracing back all the breakage, I finally arrived at the code that was actually causing it. Once I actually saw what was happening, it was easy to work out.
At some point, someone had added a cache for some expensive-to-compute data for actors in the world. The cache had a mapping of actors to cached-value matrices, and each actor had a single value that was a pointer to its cached value, or NULL if it had none. I think they were too big to store indefinitely? There was a little LRU cache. But, if an actor was destroyed, it wasn't removed from the cache. So its cached data would sit there in the LRU cache, like a little time bomb, until the thing expired, and then the last part of its removal from the cache would be... unsetting the pointer to cached data in the now-dead actor back to NULL. The actor's memory had already been freed, and so one int, somewhere at some random location in memory, would get set to 0. And once in a while, it would happen to be at a location where that would cause serious problems (usually leading quickly to a crash).
Hm... not sure it would have. Something simple like overwriting freed memory with an identifiable pattern wouldn't have helped, because by the time the crash happened, the actor's memory had been reallocated and was being used for something else.
This was on the Wii, which had some pretty bare-bones development tools, so it wouldn't have been real easy to try to use something like valgrind to do more sophisticated memory debugging or anything. It was basically single-threaded, too, BTW (I think audio and input were based on interrupt handlers), and I made it more so with the repeatable build.
Yeah. I was proud of myself. 😃 IDK how easy it is now with all these friendly toolkits, but back at that time and definitely for console development, there wasn't a lot of way around really knowing your shit. Even having a degree, it definitely helped, but even that wasn't really a full preparation for some of the difficult stuff I ran into. I wish CS prepared people better for real-world engineering; in particular I wish reading big codebases was more of the curriculum. I actually came out of school still with fairly bad working-with-production-code abilities.
Separate related story, I was disagreeing with my boss about whether to hire some guy. My boss said, but his code is really bad. I said, yeah but he's right out of school so it's to be expected. When I first graduated my code was really bad. My boss got sort of distant for a second while he was remembering, and then his face came alert again and he said, all animated, "Yeah! It was. It was terrible!" I said yeah, I know, then I learned what I was doing and it was better. It's just sort of how it works.
Yeah for sure. I'm about a third of the way through "The Art of Game Design: A Book Of Lenses" and it has proven to be a very useful book full of actionable tools to help get the creative juices flowing. The titular "Lenses" each consist of a few questions to ask yourself as you make decisions about your game. Some neat examples:
Lens 007: The Lens of Endogenous Value
A game's success hinges on the players' willingness to pretend it is important. To use this lens, think about your players' feelings about items, objects, and scoring in your game. Ask these questions:
What is valuable to the players in my game?
How can I make it more valuable to them?
What is the relationship between value in the game and the players' motivations?
Remember, the value of the items and score in the game is a direct reflection of how much the players care about succeeding in your game. By thinking about what the players really care about and why, you can often get insights about how your game can improve.
LENS 009: The Lens of the Elemental Tetrad
To use this lens, take stock of what your game is truly made of. Consider each element separately and then all of them together as a whole. Ask yourself these questions:
Is my game design using elements of all four types?
Could my design be improved by enhancing elements in one or more of the categories?
Are the four elements in harmony, reinforcing each other and working together toward a common theme?
LENS 015: The Lens of the Eight Filters
To use this lens, you must consider the many constraints your design must satisfy. You can only call your design finished when it can pass through all eight filters without requiring a change. Ask yourself these key questions:
Artistic Impulse ("Does this feel right to me?")
Demographics ("Will the intended audience like it?")
Experience Design ("Is this well designed?")
Innovation ("Is this novel?")
Business and Marketing ("Will it be profitable?")
Engineering("Is it technically possible?")
Social/Community ("Does it meet our social and community goals?")
Playtesting ("Do the playtesters like it?")
In some situations, there may be still more filters; for example, an educational game will also have to answer questions like "Does this game teach what it is supposed to?" If your design requires more filters, don't neglect them.
The author has worked on a number of experiences for Disney that he relays as well, creating bespoke hardware and incredibly immersive short experiences. His emphasis on cross-domain knowledge and attention to detail in all aspects really appeals to me.
I've had to take a little break, got some life stuff going on, but I'm hoping to finish this book by the end of the month.
As for reading list, right now it's really just a big pile of things, not much order. There's a lot of recommended reading in this book too that'll get added I'm sure. You can look at my big mess of a github repo:
There's a million issues in there with book suggestions, as well as my notes in the closed issues. I am planning to put up a little website to contain my knowledge for future reference. I'm really trying to create a habit to Learn In Public so I'm gonna put everything I do up there in that github org. Feedback and contributions of course welcome.
I am reading Tomorrow and Tomorrow and Tomorrow, a novel about game developers (or something, I'm only a few chapters in). Not very technical but I'm definitely enjoying it so far; it is a nice view into the human side of the job/lifestyle
I'm game.
I'm currently reading "Don't Make Me Think: Revised Edition" by Steve Krug. The book is primarily about website design, but anyone with half a brain could translate the design principles and main ideas into a game development context. I just finished Chapter 10(?) all about designing usability tests and how to get a feel for where the main issues are with your design.
After that, I've got "Design is Storytelling" by Ellen Lupton and "The Animator's Survival Kit" by Richard Williams queued up.
I don't think game engines make it particularly easy either. I was thinking of trying to integrate web technologies into my engine to try to ease that but it's pretty complicated
I loved the cross-platform approach Kenny has taken, it is stuff that will be useful for years, and stuff like this is tricky to find resources for. :)
Saved for when I am at the "user" stage of my game, beyond "yay saves now work" and the lower level code :)
I haven't watched yet but I tossed the transcript into claude and chat gpt
ChatGPT:
Main Idea: The best marketing tool for your game is the game itself.
Key Strategy: Develop and engage a community around your game during its development.
Understanding Steam: Learn how Steam works for both developers and players to tailor your marketing approach.
Marketing Tips: Focus on unique aspects of Steam's platform to market your game effectively.
Overall Goal: Integrate marketing efforts with game development for better results.
Claude:
Your game is your best marketing tool
Be authentic to your game and target audience
Steam offers tools to market to your unique audience
Build wishlists and community well before launch
Focus on gameplay in trailer, screenshots, description, tags
Get feedback via beta, playtests, demos; join Next Fest
Launch: Steam emails wishlisters, features in queues
Post-launch: keep engaging players via updates, discounts, events
Stay authentic and responsive to build your audience
The key is to know your game and audience, start marketing early, leverage Steam's tools, and continue nurturing your community after release. Authenticity and active engagement are crucial for success on Steam.
Game Development
Hot
This magazine is from a federated server and may be incomplete. Browse more on the original instance.