October Hackathon Session 1

We just wrapped up our sixth hackathon, and suffice it to say we had a blast! This was our biggest and best hackathon yet, and the next one won’t come soon enough.

The first day started out like every hackathon should: We took a hot tub break. Just kidding, we actually started working right off the bat. (We hit the hot tub about twenty minutes later.)

Dinner was served by a familiar face - Carl! We ate, watched the Giants game (everyone at ZocDoc is a Giants fan), then powered up Team Fortress 2, AKA TF2, AKA the best game ever, AKA why aren’t you playing TF2 right now? We played until Carson gave everyone a spy knife and the game was no longer fun.

By day two, everyone was already showing real progress with their projects. It was great to get away from all the hustle and bustle of the city and think about what goals we want to accomplish in the coming months. Hacking poolside is especially relaxing with the cool Long Island breeze and fresh air making their way through the spacious accommodations.

Around midday, the rest of our crew arrived, and we got in some Frisbee and full court basketball. Don’t think of basketball as a nerd game? The skill and trash-talking would probably surprise you. We do tire easily, though, so we put in a few more hours of hacking before some midnight TF2 and Settlers of Catan.

Most memorable exchange from day two: “Anyone interested in going to ocean?” “Nope. Freezing. So cold.” “Huh? It’s warm. Fish live there.”

Only in the Hamptons can ZocDoc devs be so free enough to even ponder the temperature of the water in their vicinity.

Day three: A new game has been invented! We dub thee, soccer tennis volleyball. (Editor’s note: This “new game” has been has been invented on at least one previous occasion. Watch for the lawsuit.) You start with a soccer ball, rally up a few devs, split up the teams on a tennis court, and proceed to play volleyball; same rules as volleyball except with a lower net and using your feet ONLY. Best game ever!

More TF2 ensued (notice a trend?). One of our devs decided his only goal was to run around with a scout and club people with the baseball bat all game every game. Now, TF2 is a strategic game with checks and balances. It requires team coordination in order to achieve the winning objectives. Imagine a random crazy person running around with the senseless goal of whacking people. It actually ended up proving quite an effective distraction and his team won. Another board game was whipped out at this point to provide an alternative to Settlers of Cataan: Dominion (a card game).

Day four: One last session of TF2 must be played before we pack up and head home. The takeaway for the weekend? The new ZocDoc class can dominate the veterans at TF2!

Spring/Summer 2012 Hackathon Report

Spring/Summer 2012 Hackathon Report

Another season, another ZocDoc Hackathon. Since the last hackathon this winter, ZocDoc’s development team has nearly doubled in size (because, of course, ZocDoc is hiring!), requiring us to split the spring/summer Hackathon into two four-day weekends. So this time, we have two Hackathon reports, one from software engineer Matt, a.k.a. “Goose”, the other from User Experience lead Chris, a.k.a. “Turbo”.

GOOSE: Hackathon starts with packing everything we need into vans and heading out to the Hack House. Packing this time around was a bit unusual thanks to the split. In any given development room, half of the people were trying to code while the other half were noisily taking apart workstations, putting everything that fit into trash bags and bubble wrapping the rest. I’m pretty sure we actually had to send Deansy out for a bubblewrap resupply.

TURBO: While the engineers bubblewrapped their 30-pound tower computers and loaded the vans (in a process known as “packathon”), the user experience design contingent, of which I am a member, cruised out to the Hamptons with nothing but our sleek MacBooks Air and a small library of inspirational design books.

Also, the UX team had the advantage of making our journey in Chris Hlavaty’s sweet, sweet van:

Now, when you think “Hackathon”, you probably think “computer programmers.” But ZocDoc’s development team is a combined group of both engineers and user experience designers. So while the developers were coding away, we were banging on Photoshop and OmniGraffle. But it’s really the same thing: we’re all just hunkering down and geeking out over an idea that we think we can kick ass on for four straight days. What’s great about bringing design and technology together is that we all have a chance to collaborate in an ad hoc way and produce even better Hackathon results.

GOOSE: Once we’ve got all the vans packed it becomes an unspoken race to get to the Hack House (first van there will get first choice in beds). Once we’re all there we quickly set up the essentials…

We packed a bunch of folding chairs to work on, but it pays to be creative. I pulled a table up against a couch and used that as a desk. (Best office chair I’ve ever used.) Then we settled down and got hacking.

TURBO: After unpacking, I got to work doing a deep dive on the UX for a really complicated new ZocDoc feature, creating flowcharts and photoshop mockups, and rapidly iterating the designs through small group reviews every few hours with my teammates, both designers and developers.

Basically, Hackathon consists of three activities: Hacking, games (of both the indoor and outdoor varieties), and eating.

The challenge of feeding twenty nerds glued to their computers isn’t a small one. Thankfully, Hack House comes equipped with two great chefs, Carl and Anna, who prepared delicious lunch and dinner for the hackathoners every day. With the food catered, we could focus on the tough problems we came to solve, while still having the energy to take breaks and clear our heads.

A few of us took a quick jaunt to the nearby beach for a dip in the cool water, and another day we checked out an estate sale next door. I thought I’d find some sweet mid-century modern furniture, but instead all I got was an inflatable crocodile to wrestle in the pool.

GOOSE: Unfortunately, there was no gator wrasslin’ at the first week, but we still had plenty of fun. Thanks to Team Fortress 2 and League of Legends, we didn’t even have to leave our coding nests during breaks. Of course, some of our more active ranks got outside and enjoyed the sun. Plenty of basketball, tennis and soccer went down. We even invented a game that was a cross between volleyball, tennis and soccer (TURBO: Which ZocDoc CTO Nick Ganju quickly mastered).

Hack House has a pool table in the basement which strangely had 25 balls. Naturally, we had some intense games of 25-ball cut throat. Table Tennis, our company’s unofficial pastime, was also played. Some of us even looked up at night and saw the big dipster.

TURBO: After much weaving of code and pushing of pixels, and a lot of kicking of soccer balls over a tennis court net, we presented our projects to each other. Our teams focused a lot on improving our internal tools, especially on some sweet data visualization tools to help us keep a better eye on the pulse of ZocDoc. For my project, we were able to make a lot of progress by changing the design pretty radically a few times, but I still had time to hunker down and make a fairly polished finished product by the end.

![]

Concurrent Current: Concurrency’s Currency’s as Current as Currants

You like concurrency, right? Everybody likes concurrency. More and more stuff is happening at once – and when a $300 laptop comes with 16 cores, it’s just a waste to not put them to good use.

Well, there’s good news and bad news. The good news is that .NET 4.0 comes with a sweet set of collections and classes that tremendously help with handling concurrency. You just declare a concurrentInsertYouCollectionHere variable, and it promises to handle your threadsafety correctly. That counts for a lot in web world, when you’re getting a million hits per minute, your users are posting videos to your timeline, patients are booking doctor appointments, doctors are updating their schedules, etc. It’s awesome, right?

Right – but concurrency comes with costs. That’s the bad news. Handling concurrency is hard. Not even NP-hard. Just super-hard. It can elude the greatest minds when handled carelessly (see the infamous heisen-bugs). It’s just not something you want to code unprepared for.

So harken, children, to the tale of ConcurrentDictionary. It’s the story a hero with a dark side. At ZocDoc, we use ConcurrentDictionary a lot. It’s a lifesaver sometimes. You can declare a static ConcurrentDictionary of things, and add to it, modify it, get from it in a thread-safe manner. Our internal caching tools leverage this class pretty extensively.

Sometimes we have a two-part key. Let’s say (int, int) or (long, long), but we need access to all keys with the first part, too. In short, we have something like this:

ConcurrentDictionary<int, ConcurrentDictionary<int, TValue> > two level index.

It performed well for a long time (since we migrated to .NET 4.0) but eventually we started noticing glitches in GC activity on our production servers. Obviously, we’ve been growing at a crazy pace, and we’ve always given GC a hard time – but it always caught up. Then, suddenly, it started coughing more often than usual. Dr. House wasn’t around to give us a deus ex machine diagnosis, so we had to investigate.

As it turned out, we had a problem with our dumps. Err rather, taking and analyzing several particularly large dumps revealed the problem. Our preferred divination tool, WinDbg, revealed we had way too many GEN2 objects. The poor garbage collector had to walk a massive graph of objects to do its job – and what it was really choking on were the objects in the Large Object Heap, which – albeit short-lived – are born old (like Gen2 objects and certain East Asians).

For some reason we had a lot of objects in Gen2 – a lot of objects we hadn’t realized existed. And they were just objects or object arrays. (Weird!) As usual, we dug in deeper and discovered the one to blame. His name was ConcurrentDictionary. The knight in shining armor turned out to have some nasty gangrene under the hood.

ConcurrentDictionary allocates an array of 4* concurrencyLevel objects that are used for synchronization purposes. The default level of concurrency (when you don’t pass it in the constructor) is equal to the number of logical processing cores on the machine. So your regular 4 core Xeon allocates (at two threads per core) an array full of 32 objects. And what if you have 1.5 million of those dictionaries? Smooth move, buddy – you just added 48 million objects for a GC to walk. And your concurrency level is definitely not 48 million…

Clearly, we had to come up with a better solution. One way would be to take a database approach and store everything in a sorted dictionary (e.g. red / black tree), sacrificing a little bit of speed for better memory efficiency and range queries.

But when we examined our codebase, it turned out we didn’t really need range queries (apart from occasional time searching). What we needed was a hashtable of hashsets / lists / dictionaries. Sometimes we can get away with ConcurrentDictionary, TValue> if we don’t need access by the first part of the key. Tuples are neat, but they are also classes. If you write your own tuple as a struct with the same behavior as Tuple, you will gain a little extra speed and GC efficiency. This is especially true when using it with regular dictionary (which uses structs as internal storage, as opposed to classes in ConcurrentDictionary).

Eventually, we settled on a technique called lock striping. Instead of allocating one object for locking purposes, we now allocate an array of those and partition the set of accessible elements into roughly equivalent subsets. This means that one object in the “Lock array” is responsible for only parts of the information stored in the data structure.

Introducing

ConcurrentTwoLevelDictionary<TKey1, TKey2, TValue>
   where TKey1 : IEquatable<TKey1>, IComparable<TKey1>
   where TKey2 : IEquatable<TKey2>, IComparable<TKey2>

It offers all the methods you would expect from a decent dictionary interface, and it also covers overloaded methods of concurrent dictionary like:

public void Add(TKey1 key1, TKey2 key2, TValue value)
public void Add(STuple<TKey1, TKey2>, TValue> item)
public bool ContainsKey(TKey1 key1, TKey2 key2)
public bool TryGetValue(TKey1 key1, TKey2 key2, out TValue value)
public bool TryUpdate(TKey1 key1, TKey2 key2, TValue newValue, TValue comparisonValue)
public TValue AddOrUpdate(TKey1 key1, TKey2 key2, Func<TKey1, TKey2, TValue> addFactory, Func<TKey1, TKey2, TValue, TValue> updateFactory)

But to really make our lives easier, we built methods that utilize only the first part of the key, like this:

public IEnumerable<TValue> this[TKey1 key1]
public bool Remove(TKey1 key)
public bool ContainsKey(TKey1 key)

The internal implementation uses ConcurrentDictionary at the top level and regular dictionaries in the lower levels. Each operation (apart from reads and writes to the top level which are atomic i.e. provided by concurrent dictionary) uses something like this:

lock (locks[GetLockNumber(key1)])
{
    \\stuff
}

The GetLockNumber function returns an integer from the range [0,locks.Length). We use key1 hashfunction to give us a roughly equivalent distribution of locks, and to ensure that accesses to the same key take the same lock.

Occasionally, we need to utilize a function like this:

private void AcquireAllLocks()
{
        for (int i = 0; i < locks.Length; i++)
        {
            Monitor.Enter(this.locks[i]);
        }
}

…and ReleaseAllLocks equivalent. Those operations are needed when we clear out the dictionary, get a Count, or perform other tasks like GetKeys that operate on the whole dictionary. And that’s it! We replaced a few instances of ConcurrentDictionary> and we killed, oh, 50 million objects on our object Gen2 list. (The fewer objects there, the better.) Hooray, concurrency FTW!

GOTO Where You Need To Go

[Happy April Fools’ Day! We hope you liked our little joke.]

You’re sitting at your desk, coding along, when you find yourself in this pickle: you need to reuse some code. You’ve got a nice little snippet that does exactly what you want, but it’s already being used in some other method. You stare longingly at that piece of code drooling and mumbling, “I want to go to there.”

Why not just goto that code? Have we forgotten our roots? Have we forsaken what some might call literally the oldest trick in the book? We at ZocDoc haven’t forgotten where we came from. So we started using gotos. At first just a little. Then we got hooked on the sweet, sweet nectar that is the goto statement. Then we started thinking…

We have lots of new developers starting in June and C# has some really complicated syntax. There are for loops, while loops, foreach loops, and recursion. How many different kinds of loops do you really need?! Then there are function calls, classes, inheritance, etc. You shouldn’t have to be an expert programmer just to do a simple thing like adding two numbers and printing the output to the console!

Using goto solves these problems. It simplifies the language, cuts down on training time, and make your codebase easier to understand. It’s a no-brainer! The advantages of this new system are endless. Our code is less indented. Our code is flexible, a goto can go anywhere in the current scope. No need for parameters – everything is statically shared. Oh, and say goodbye to nasty stackoverflow exceptions! Our goto pattern is marginally faster than regular method calls and it is immune to stackoverflows. (See our code examples below)

Additionally, we’ve found a plugin (goto.js) to add goto functionality to javascript, which means that developers now only have to learn one language. You can literally copy and paste back-end code into the front-end!

Some developers were worried that we’d abandon some of our most popular coding conventions when we made this switch. Fear not, devs. We still fully encourage the use of x1, x2, x3, x4 as label names for your gotos to stop the proliferation of those hard to read prose-y names.

Unfortunately we weren’t able to get rid of all if statements and function calls, but we did succeed in getting rid of most control structures. If only C# had a comefrom statement, we could get rid of function calls! Despite this obvious limitation, we still feel this discovery is a big win… So much so that we are calling this simplified language C## (some of us are calling it Dmaj, there isn’t a clear winner yet).

Some people say that goto is bad, or makes your code “hard to follow.” Well, why do you think breakpoints were invented? If goto is good enough for the linux kernel, it’s good enough to use at ZocDoc. What can we say?

goto-it people!

For Loops the Archaic Way

    for (int i = 0; i < 10; i++) 
    {
        //wow it's so hard to remember all this pesky syntax. 
        //  was it semicolons? 
        //  how many? 
        //  what order do the expressions go in? 
        //  omg!
        Console.WriteLine(i);
    }  

… and Simplified to use goto

    int i = 0;
    x1:
    Console.WriteLine(i);
    i++;
    if (i < 10)
    {
        goto x1;
    }

Fancy OO-Style Inheritance

class Greeter
{
    public virtual void SayHello(Friend f)
    {
        f.HighFive();
        Console.WriteLine("hello");
    }
}

class FrenchGreeter : Greeter
{
    public override void SayHello(Friend f)
    {
        f.Kisses();
        Console.WriteLine("Bonjour!");
    }
}

…Simplified to use goto

void Greetings()
{
    Console.WriteLine("are you french? y/n");
    bool isFrench = bool.Parse(Console.ReadLine());

    goto greet;

    franch:
    //because c# doesn't have a comefrom statement, 
    //    it's not practical to use goto:kisses;
    //    the most pragmatic thing for us to do was to allow function calls 
    //    in very limited circumstances.
    //    however, comefrom syntax would look like this:
    //        goto: kisses;
    //        comefrom: kisses;

    kisses();
    Console.WriteLine("Bonjour");
    goto done;

    mercan:
    highFives();
    Console.WriteLine("Hello");
    goto done;

    greet:
    if (isFrench)
    {
        goto franch;
    }
    else
    {
        goto mercan;
    }

    done:
    Console.WriteLine("greetings have concluded");
}  

Winter Hackathon

Every few months the ZocDoc dev team has a weekend hackathon for developers to drink and play team fortress 2 work on really cool projects outside the scope of everyday tasks. Because hackathon is offsite, it starts with packathon. We pack up tables, chairs, food, drinks, computers and monitors. Some of us (Dan) tend to be extra careful with our packing.

With everything safely stored in cargo vans, we buckled up and headed for the Hamptons. Four hours and multiple Crunchwraps Supreme later, we pulled into an icy driveway; the temperature couldn’t have been above 10 degrees.

But our chef Carl had beat us to the house by a few hours, so we walked inside to find a bowl of freshly made guacamole and the smell of a steak dinner cooking. We unpacked everything in record time, and (once again) transformed an elegant living room into a hacker’s paradise, setting up folding tables and filling every room with monitors.

With all of the developers crammed into one room, the house started buzzing with project brainstorming sessions, the sound of aggressive typing and some nice jams courtesy of turntable.fm. A lot of us worked on infrastructure projects that weren’t necessarily glamorous, but make us nerds happy and our web servers happier (more to come on performance improvements in a future post).

Some of us worked on new internal tools for our sales, marketing, and operations teams. Others worked on a fun Kinect-powered map display of searches and/or appointments (which you can move or zoom with hand gestures), that will run on a TV in our New York office. Since ZocDoc has some awesome data, we love to build tools for visualizing it, so a few developers worked on a new generic graphing tool that can filter, group, split, aggregate, and display any data set. The data visualization application does all of its work client side with javascript, so you can manipulate your data faster than a lightning kick.

But the real work began around 10 PM every night, when we fired up TF2. Though we’re all “engineers” at ZocDoc, we found that only a handful of us are true engineers. The rest can’t resist the glory of flamethrowing someone’s face off or ubercharging into a melee of BLU to capture the last point. Quick protip for the TF2 rookies (cough, cough, Carson cough): when you see an Ubercharge coming your way, aim for the medic’s feet. You can launch him into the air, and if he flies far enough he’ll lose his lock on the heavy – then it’s a free for all.

A few of us even braved the cold to play some basketball on the court in the front yard, and we also organized a massive four-hour game of poker. (This is important, because every developer secretly believes he could win the World Series of Poker if he played seriously for a few months.)

As hackathon starts with packathon, so it ends with packathon. On Sunday, we packed up the NASA control center we’d created and returned the Hamptons mansion to its natural state. We headed back the city – all slightly better at TF2, slightly worse at sleeping, and ready to implement the awesome projects we’d come up with at WNTRHCKTHN 2012.

How To Win With Chun-Li

On the ZocDoc dev team, we have two video games that are kind of a big deal. First, there’s Team Fortress 2. TF2 is fun, but we don’t tend to take it that seriously – it’s mostly a relaxing activity at the end of a day of hacking. Then there’s Street Fighter II: Champion Edition. This game is serious business. Nick is the definite champion (at least when he’s playing Ken), but competition among the rest of us can get fierce.

I’m here today to talk about Chun-Li, the lone female character in this version of the game. She’s my strongest character, and she can be underappreciated. Ken is the “default” character, Ryu is a bit faster, Bison is haxx, and Dhalsim has the flashy moves. But Chun-Li is fast and has the moves to keep your opponents off-balance, if you use them unpredictably enough. According to Wikipedia, she’s also the first playable female character in a fighting game.

Here are a few of my favorite techniques:

• Jump in with a short or medium kick. (If you use medium, make sure you’re not pressing down at the time – otherwise you’ll do a head-stomp.) Follow it up with one of these:
  • A throw (towards and fierce) – works even through a block.
  • A low kick into lightning kick – usually they’ll block, but lightning kick does a lot of damage through block. And if they miss the block, you can chew up half their health this way.
  • A neck-breaker (the kick where you flip over them, which is towards and roundhouse)
• Neck-breaker to knock them down, then neck-breaker again as they get up. Repeat until they remember which direction you have to hold to block the neck-breaker, then throw when they’ve blocked it.
• Air-throw. This is a very powerful move, if you have the reflexes to pull it off. All you have to do is towards + punch when you’re near them in the air. This can also be used if you’re stuck in a corner and they do a jumping attack – just jump straight up and toss them to the ground.
• Note that the head-stomp (down + medium kick in the air) goes through several attacks that normally damage anyone that touches them, such as Blanka’s electricity and Bison’s psycho crusher. It doesn’t do much damage, but it can throw the opponent off-balance.
• Special note for facing Ken / Ryu / Sagat: try to walk towards them to lure them into doing a dragon punch, but back out just in time. When they come down from the dragon punch, throw them.

Overall, stay in the air a lot (except when facing Blanka, Vega, or Guile), focus on your kick and your range, and try to deny them the opportunity to touch you. Don’t neglect your throws.

Now go for that perfect!

Measuring and Optimizing SQL Performance

At ZocDoc, we like performance. A lot. (We also like performance alot. He’s the one wearing tap shoes.) Performance was an especially important word for us during 2011. We expanded across the country and saw an increase in search volume by several orders of magnitude – while managing to reduce the time of our search algorithm by orders of magnitude (and made it smarter to boot).

But scaling this rapidly wasn’t entirely seamless. While few of our public-facing pages even touch SQL Server, we have many internal pages and processes that do. All this means that the database (which talks with several web servers and other applications) is our main obstacle to horizontal scalability. Unfortunately, your ability to optimize database code is only as good as your ability to measure it. And ZocDoc’s measurement tools just weren’t good enough.

That’s why we developed ZDSqlCommand. SqlCommand is a class in the .NET framework that is responsible for talking to Sql Server. ZDSqlCommand wraps it, exposes the functions on SqlCommand that people use, and does some tracking for each.

Let’s get a little nitty-gritty. If you introduce a wrap like ZDSqlCommand, you now need everyone to use it, right? That turns out to be easy when you have SqlHelper (ZocDoc’s micro-ORM that uses ZDSqlCommand under the hood). ZDSqlCommand is a drop-in replacement for SqlCommand, so it was simple to update our remaining non-SqlHelper code that was written with SqlCommands.

Finally, we set up FxCop and created a rule so that all “new SqlCommand()s” break the build. If you break the build, you have to put a dollar in Allen’s Steve McQueen lunchbox. That lunchbox is going to pay for a sweet keg party someday, or so he says. Anyway, all code that runs SQL statements in ZocDoc now goes through ZDSqlCommand in one way or another. Problem solved!

How does ZDSqlCommand help us measure and optimize? For each method on SqlCommand, ZDSqlCommand creates an object called SqlLogItem which records how long the command took to execute, the stack trace, the query text, when the query was executed, bytes sent and received, and the parameters. Note that stack trace (with line numbers) uniquely identifies the web page and code path that invoked a SQL statement.

///example of a ZDSqlCommand method
public object ExecuteScalar()
{
    var stack = GetAndCheckStackTrace();
    ResetConnectionStats();
    var sw = new Stopwatch();
    sw.Start();

    //delegate to the SqlCommand
    var scalar = mySqlCommand.ExecuteScalar();
    sw.Stop();

    var connStats = connection.RetrieveStatistics();
    SqlLogItem.Record(this, sw.Elapsed, connStats, stack);

    return scalar;
}

And thus begins the long, strange journey of SqlLogItem. We then serialize it to disk for offline processing, and store it in a list in the HttpContext.Items collection (a key value store tied to a single web request). On the Request_End event, we check for this list and do two things with it.

First, we check if anyone called a query inside a loop. This is trivial: if 5 or more log items have the same stack trace and their query text doesn’t contain Insert or Update (which can legitimately be called from a loop), it’s deemed a query in a loop. If this happens, the entire dev team gets an email and someone goes and fixes it.

Second, each web server has a static Dictionary of SqlStat objects. These are the aggregates of all the individual log items, keyed by their stack traces. SqlStat is an object with TotalCalls, TotalQueryDuration, StackTrace, and QueryText. The most useful report is simply to sort these by TotalQueryDuration. Because many of the same queries are run over and over through the day, TotalQueryDuration typically increases over time, and it’s difficult to figure out if a query is running normally or spiraling out of control. So we divide TotalQueryDuration by ServerUptime to get time percentage, which is a more useful metric. If the query runs at the same rate throughout the day, the percentage will stay relatively constant. If it runs at the same rate but begins executing slowly (for whatever reason), the percentage will go up. If the query is really heavy but only runs when you first boot the server, the percentage amortizes over the day.

You can see here that each column is a different web server. Each row is a different query, and the top of each column is the total for that server. At the very top we sum up the values for all the web servers and show the overall totals. The big number is the percent time (query time / server uptime), which can go over 100 because web servers are multi-threaded. To optimize, we simply work our way down the list. Because we have a lot of expensive queries that only run once at startup, our graph looks a little crazy for the first 30 minutes after booting a server, but afterwards it stabilizes and shows an accurate picture of where the DB load is coming from. This page is displayed on the dev room wall-monitors, and it automatically refreshes every minute.

Additionally, when any query goes over 10 percent, it turns red on this chart and our team gets a text message about it. This has come in handy a few times. Once, we found that SqlServer wasn’t using the correct index for a certain query because that index’s statistics were skewed by an unusual load pattern. Another time, we realized an automated process was running a heavy command once a minute instead of once every 30 minutes.

Finally, we get a nightly email of the stats page in case we ever want to know how a query has behaved historically. It’s boss! This is optimization at its finest – leafing through data, finding the kinks, and making everything work a little smoother. It’s so easy a dog could do it!

Barkley keeping an eye on the SQL stats