Data scraping
Ok, I quit my job like 2 months ago because I couldn't stand it anymore. We were promised that we'd be given full support to do everything legit on a project website, but then 6 months into the project we were still being asked to scrape data from other sites of the same content.
My question is, do many sites do this too? Is it a common practice? Is this ethical. I know that several sites have similar posts, seems like it's being taken from another source.
That project I was working on was a job site, and I hated the fact that we were getting `our` jobs from other sites, feels really stupid and illegal, although the sites we got data from didn't seem to stop my crawlers from crawling there pages. It would've been really obvious given the number of pages accessed sequentially everyday, although I did put some random pauses in between.