Crossing into Canada is no problem.
Getting back into the U.S. can be zoological, even with a U.S. passport. A few years ago I took a wrong turn in Blaine, WA, and found myself at Canadian customs. They were quite helpful and directed me into the turn-around lane to the U.S., where I was taken into a room for questioning while my car was searched. I got the distinct impression that the U.S. customs agents were bored to tears.