Poll of the Day > I'm currently scraping all projects on CodePlex

Topic List
Page List: 1
Yellow
02/25/21 9:47:36 PM
#1:


To hand over to internet preservationists.

Can't believe no one has done this yet. Someone has to!

Should end up being 1.5 terabytes in total. Yes, every project on CodePlex collectively takes up 1.5 terabytes. All 108508 projects. It should also be done in a little over 24 hours, assuming I don't get IP banned with my 1-second delay between downloads. (My VPN IP)

Their API isn't public, so I had to sniff it out and reverse engineer it a little.

This has made me feel pretty lonely. It's one of the most important things I've done in my life, yet no one irl understands or even cares.

It is a new record for me, going from concept to result in 3 days. Scrapers are always pretty easy. The caveat is that after I run it once, it's completely useless.

---
why am I even here
... Copied to Clipboard!
Topic List
Page List: 1