RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training