18 March 2019

Here is the PDF of my deck from the talk!


“With great power comes great responsibility” -Voltaire / Spider-Man / Stan Lee

  • This talk gives high level best practices of releasing new features
  • All features don’t necessarily need to follow all guidelines
  • Any feature needs some combination of these


  • Gamedev Tools Engineer since 2003
  • Infinity Ward, Kojima Productions, Insomniac
  • Leading a tools team at IW
    • We are hiring, and having a party tonight!
    • tinyurl.com/toolshappyhour2019 for details
  • History in revision control, branching, file formats
  • Built asset editors, level editors, tools frameworks

^ But this talk isn’t really about me it’s mostly about…

My Evil Twin

  • Extensive experience in breaking artists & designers
  • I’ve halted entire studios’ progress for hours
  • I’ve degraded artists’ productivity for days
  • I’ve released tools that are:
    • undertested (buggy)
    • underdesigned (don’t work in practice)
    • overdesigned (confusing/too many features)

^ No details b/c of Vault, talk to me later tonight

^ All this qualifies me to lecture you about how to not be like me, and waste people’s time


^ You may or may not have consciously chosen to be here

^ You have written some code, decided things should be different

^ Or maybe you are following up on a feature request opened by a user


^ Stop right there, buddy.

^ You almost definitely aren’t ready to pull the trigger.

^ I’m here to tell you all the things you need to do (to not be like me).

1 - Question Your Design

  • Have you actually done all the design work to ship this feature?
    • Remember: impact usually surpasses intent
    • Take a step back and make intent crystal clear
  • What problem are you trying to solve?
    • Is this something that only bothers (or makes sense to) you?
    • Imagine giving an “elevator pitch” of explaining this design
    • Do you sound like a crazy person?
    • Is this a waste of time/effort?

^ Try your best to talk yourself out of making the change

^ It’s perfectly ok to start a lot of changes and walk away!

^ When in doubt: put it on ice and move on to the next thing

2 - Estimate The Impact

  • Have you considered how it will impact:
    • Every workflow permutation (use case)?
    • How do you know? Have you searched the code (GREP)?
    • How will offsite staff and outsource vendors work?
    • Downstream projects (teams offset in time)?
    • Fellow engineers (different from users)?

^ A good time to do some “Rubber Duck Debugging”-style explainer of the change

^ A preliminary code review to help make the impact knowable

^ Just looking to understand the scope of your change

3 - Perform And Document Your Testing

  • The bigger the impact, the more testing you need

  • Be greedy with automated tests
    • No cheating: actually wait for them to finish!
  • Critical code paths should be stepped through in the debugger
    • Don’t end up saying “How did this ever work?”
  • Document testing you have done in change comment
    • Adds value to the code review process

^ Remind code reviewers that your code works by stating tests

^ They aren’t strictly looking for bugs, evaluating your approach

^ If tests are documented reviewers may think of tests you are missing

4 - Measure And Document Your Performance

  • Is it faster? Is it slower? By how much?

  • What is the bottleneck? Did you change it?

  • Have you measured the impact on resources?
    • CPU: are you wasting CPU cycles needlessly?
    • Memory: what about going wider on many-core CPUs?
  • Disk Space
    • Have you considered how disk caches will be evicted?
    • How will you know if eviction has a bug?
  • Server Load
    • Do you have caching (or rate limiters) on your key API routes?

^ You should know and document all these things

^ Take credit for speed improvements, and justify perf hits

^ Demonstrate through documentation that you have thought it through

5 - Prepare For Failure

  • What does the worst possible failure look like?
    • How (and when) will you know if this is happening?
  • How easy is this change to roll back?
    • Are there costs that make roll back painful?
    • If so, what is the commensurate change in testing?
    • Should you launch this feature as a features toggle (setting)?
  • Are you actually prepared to roll back if it causes a problem?
    • What time is it right now?
    • When do you plan on leaving work?
  • How big of a problem are you willing fixing w/o roll back?

^ The best practice for releasing a change is to roll back at first sight of defect

^ Engineers, however, are in the headspace of thinking through the technology

^ Roll back first trains your muscle memory to respect others’ time more than your own

^ Train yourself to roll back at the first sign of trouble

6 - Update User Documentation

  • Have you searched for documentation that needs updating?

  • There may be many places where “documentation” exists…

    • Wiki
    • Code comments
    • Documents in revision control
    • Recent message threads about an issue

^ Documentation falls prey to broken windows effect, fight it

^ Pointing to relevant documentation when people ask should feel great!

7 - Send Good Notifications

  • Now the feature is release, it’s time to tell people!
    • Always TLDR in your emails
    • Provide details for technical stakeholders (maybe separately)
    • If the performance wins/costs are substantial, tell people
    • Include images, GIFs, videos: eye candy helps!
  • Consider and advise about issues you may expect
    • Include some advice for fixing potential issues
    • Be on call to answer replies within a time window

^ This is tricky to get right

^ Good communication is easy to lose and hard to gain

^ You have to make it short and simple to have a shot at keeping people reading

8 - Do The Follow Up

  • Contact users that are meant to benefit from the change
    • Did they actually pay attention to your notification?
    • Did they actually benefit as you intended?
    • Did they experience a bug, and forget to tell someone?

^ This is critical to you improving as a developer

^ Any missed communication at this stage means you need new procedures

^ Remember that different users receive communcation in different ways

^ You may need to reiterate the same information multiple ways

9 - Post Mortem Yourself

  • How many times did you have to roll back?
    • How are roll backs trending?
    • What can you do to decrease it?
  • Verify with people randomly:
    • Your TLDR was short enough
    • Your details are pertinent

^ I like to do this in the kitchen, just ask if people saw and understood my notes


  1. Question Your Design
  2. Estimate The Impact
  3. Perform And Document Your Testing
  4. Measure And Document Your Performance
  5. Prepare For Failure
  6. Update User Documentation
  7. Send Rich Notifications
  8. Do The Follow Up
  9. Post Mortem Yourself

^ Now you are actually done shipping your feature. It’s a lot of work:

[.build-lists: false]


Twitter: @gorlak


  • Twitter: @thetoolsmiths
  • Website: thetoolsmiths.com
  • Chat: thetoolsmiths.slack.com

Call of Duty Happy Hour: tinyurl.com/toolshappyhour2019

See you later tonight!