For trains and buses, as far in advance as possible to ensure availability. For flights, there’s a “sweet spot” in many countries between paying too much for advance and too much for last-minute. Check the airline web sites frequently for special offers. You could save a small fortune with several of them if you’re prepared to schedule your trip around discounts.
In many countries, the ones the ATM shoves out at you are far too large a denomination for everyday use. I found this to be an issue in Bangladesh, Vietnam and India for a start. Many banks will change these for no fee, so shop around if the first one you go to tries to charge you. Better to have a brick of small change than have to overpay every rickshaw driver who, funnily enough, doesn’t have enough change and has to force himself to accept a huge tip.
Many countries charge a fee to leave depending on whether you’re exiting by air, land or sea. These charges can vary on where you’re from and not all airlines/countries add them automatically to your tickets, so you’ll need to have enough currency with you at your departure point to pay. As examples, Oz add it to air tickets as standard; NZ don’t but have handy ATM-style machines to make payment on plastic very simple; Vietnam requires payment by cash in Dong or US Dollars and their charge varies dependant on your nationality.
Check ATM availability if you’re going to rely on them. If in doubt, load up on currency when you can and make large withdrawals when you do to lessen the effect of charges (also see the tip about opening a Nationwide account).
Similarly to buying goods, find out from a local how much your taxi fare should be. Again, if you’re being quoted too high a price, walk away. There are hundreds more taxi drivers who’ll gladly take you. Always agree the fare in advance before getting into the cab.