Yup, gonna say get familiar with your language of choice. Familiar with its prepackaged libraries so that you don't reinvent the wheel, code, test the code and play with it. Test its limits.
Web crawlers, huh? Learn some internet programming and networking with your language of choice. Better that it is a scripting language.