Yes! You need calcium and potassium to prevent a charley horse, and milk and bananas have those nutrients.
Also, keep a bottle of electrolyte fortified water on the bedside table if you get thirsty at night, because that will also help prevent depletion of electrolytes, which can result in a charley horse.
If you get a charley horse, flex your foot and relax your body, keeping the foot of the affected leg flexed for a few minutes while you breathe normally. Then gradually unflex the foot. That will lessen the severity.