Therefore, I make sure I can always monitor without relying on audio routed through the DAW. Larger buffers mean more samples of latency.
In practice, I have always valued reliable recording, which means I want larger buffers, since a buffer underrun is a gap in the recording.
Total round-trip latency for my current settings are about 12ms, which is not fast enough to record with.
On my computer, I run the buffer for Ableton Live at 128 samples, which at 44.1 kHz is over 6 ms of latency, and that's one-way. At a 96 kHz sampling rate, that amounts to less than 1 ms. RME, for example, advertises a latency for their BabyFace interface as low as 14 samples for OS X drivers. If any plug-ins are part of the signal chain, it gets a lot messier, with some plug-ins introducing 500ms or more of latency (that's rare for most plug-ins but possible or even likely for look-ahead peak limiters or convolution reverbs).
Latency is introduced by the analog to digital conversion process, and then by several different buffers, such as the bus buffer (e.g., the USB buffer) and usually at least two software buffers (from the driver and the DAW). There are many variables that affect total latency because there are several sources of latency.
