OBJECTIVE: Develop an automated application to scrap/mine data ongoing from a set of specified websites in pre-defined criteria. And preparing the information in an agreed format for end-user allowing custom filtering on demand. Project Scope: Phase 1 : Create crawler scripts for data extraction of a specified criteria. This will run on multiple sites if required. Phase 2: Process and map the data in an easy to use required format for data base (data mapping) Phase 3 : Populate the data in database of the application and make it accessible to end-users You should be able to plan the following: 1. Data collected from different websites may be in varied formats. How would you standardise the format to make it usable and populate it in a centralised database? 2. How would you deal with sites that may employ extra measure to block web crawlers, bots. 3. How do you verify the quality and accuracy of the collected data Skills (strictly required): At least 5 years of experience in creating web crawlers for data scrapping Experienced developer for automated data scraping/mining/extraction programs or apps Developer with experience in data processing and formatting Switching Van IP Address may be required for crawling certain websites. Experience in this is a must. Excellent command in English communication Extensive Experience in Programming, PHP, Javascript, MySQL, Data base designing, Web applications, servers. Apply with the following: 1. Details of previous data mining/web scrapping projects you've worked on 2. List of all data mining application, softwares, tool you've used 3. Your Resume/CV ------------------------------------------ Added 8 FEB 2012, 13:07 PM EST We will only work with Individuals NOT ASSOCIATED with any Company/TEAMS/Firms/Groups Please apply accordingly. Desired Skills: Data Structures, PHP, Data extraction, Web crawling, data mining Keywords: Database Development, mining, Data, Web, PHP, Structure, crawling, extraction