The trick comes in the "almost" part. It couldn't really be the case that you'd pick up exactly as much speed as would be needed to reach the same spot on the other side. Some energy always finds a way to escape.
Think of swinging on a swing. If you just push off once and then sit still, the swing doesn't keep going back and forth forever. Gradually it slows down.
It's the same situation when you step into that hole. Down you go, and when you come out the other side, your feet will stop at slightly less than ground-level. If you miss your chance to grab onto something, you will fall though the entire planet again. Back at your starting place you rise slightly less than last time, and so on.
See the trouble? It would be a fast way to travel, but miss your step and you will wind up yo-yoing inside the tunnel, each time rising slightly less than the time before. Eventually you will stop moving altogether and be stuck in the middle -- with a long climb home.