Candidate overview Significant expertise in rapidly crawling sites and comprehensively cataloging the pages of a site. Experience with Scrapy or similar frameworks a plus. Project overview Input: a) list of sites (can be provided on ad-hoc basis) Output per input site: a) Site name b) # of pages indexed c) list of outbound links per page indexed (note -- the links should be within the body of the site content -- headers, footers, ads should not be counted) c) summary of outbound links from site per destination domain Desired Skills: Python Scripts & Utilities Crawling Indexing Spidering Keywords: Web Programming