Warmer temperatures and more periods of drought are making it increasingly difficult to grow and harvest the iconic crop.