Granted you punch a hole in a brick wall, nothing happens to you as a direct result of punching it.
However it's eventually found out that you did it and you get sued for damage, you lose all your money and everything you own, leaving you broke and homeless.
I wish I could time travel without causing paradoxes or destroying time.