Is a golfer always responsible for damages caused by their errant golf shot?

There's a debate in my office, some say you're responsible for the damages caused (off the course) by a bad shot, and others say if it's an accident you're not responsible.