After a quick call to Nest support (only spent 10 minutes on hold) and speaking with one of the support guys, I was able to get a new base plate sent out to me which seems to be staying cool. The support guy mentioned that the FETs in the base plate can go bad and create a short, which can be confirmed by moving some of the wires around. SInce I didn't have the Nest mounted at the time, we couldn't test it out.
Installed the new base plate and left it for a few minutes. It stayed cool, so it looks like the new base plate solved the problem. Hopefully we don't run into any more problems with the Nest.
Pro tip: If you're going to replace a thermostat with a Nest, keep the old one around. Might come in handy as a temporary replacement if the Nest goes down for any reason.