Dogs dig for many reasons, but the main reasons for the behavior go back long before dogs were even dogs, to their wolf ancestors. With this in mind, one can argue that digging is as much a part of being a dog as barking or sniffing.
We know from science class that heat rises, right? So, it could be that perhaps your puppy is trying to create a cool spot for them to take a nap by digging a hole beneath the ground, in the cold and comfy dirt.
As we touched on earlier, dogs sometimes will dig and get under barriers, like fences. The reason behind this is that they might be trying to find more exciting locations, look for food, or even a mate that they think exists outside of your yard.