At what point do I need to worry about improving my credit score?

You dont ever need to worry about it. Get a credit card, put your regular expenses on it, and pay it off every month. Pay all your other bills on time, and your credit score will be fine. Your credit score will take care of itself if you are responsible with your finances.