We have always captured lots of deer mice in our kitchen under the sink in a metal, box-like live trap that I use.  I would capture the little critter, throw it out the back door, and reset the trap.  The next day there would invariably be another occupant in the box.  One day my young son suggested we mark the mouse before tossing it outside, so we placed a spot of blue paint on the tip of its tail.  The next day when I checked the trap under the sink, it contained a mouse, you guessed it, with a blue tail.  From then on, I took the mouse a couple of miles down the road before releasing it on my way to work.

This species actually makes a pretty good pet.  Very cute and very docile.  But mouse droppings in the kitchen are not acceptable.  Plus, this species is implicated in carrying the dreaded Hanta virus.