Thread replies: 5
Thread images: 1
Thread images: 1
File: web-scraping-services.png (42KB, 658x224px) Image search:
[Google]
42KB, 658x224px
Is there a program available to go through every one of my bookmarks and download the FLV of any video or (say .mp3) for music file and download it?
From my limited understanding, I need a webscraper and somehow to have it only download video and audio files from it. I will use premade programs now, but in the future what programming languages would be useful for this sort of thing as well as webcrawling?
Second question: I am learning to code a bit. I know some html, xml, lua, and java, vhdl and matlab. For something like I would like above, what would be the ideal programming languages to learn?
I will finish Java, move onto node.js, Python, objective C, and C++. I need to learn R and PHP and for general website scraping and collecting data + a sql or nonsql database systtem. Possibly Go and Kotlin.
I am Engineering so I need to know some code and I've just been to lazy sit down and work on it. Ideally, I'd like to figure out really what differentiates each language from one another, which features each has and does not have, and not learn languages that are redundant. Is there a good website to learn this information from?
My thinking of finishing Java then Python then Objective C then C++ to be the easiest route and let me work and finish some simple projects first.
Last question that ties into the first:
While I am in University, I have access to ~7 academic databases. Illegal yes, but how would I theoretically be able to download as many studies/articles/published material from it as possible? Kind of ties into the first.
>>
> While I am in University, I have access to ~7 academic databases. Illegal yes, but how would I theoretically be able to download as many studies/articles/published material from it as possible? Kind of ties into the first.
They will notice, block you and then possibly prosecute. Don't do this.
Honestly, you could do a 95% solution with wget working from your bookmarks exported as html.
Java and Lua are fine for this, but it doesn't play to those languages' strengths.
>>
>>60872779
>While I am in University, I have access to ~7 academic databases. Illegal yes, but how would I theoretically be able to download as many studies/articles/published material from it as possible? Kind of ties into the first.
I have no idea if universities actually watch for this stuff, but in the interest of staying out of jail I would assume they do. Assuming that, this is exceptionally difficult. You could spend months trying to step everything up to not get caught doing this.
Basically, to do it safely, you'd need to steal somebody else's credentials and then do it over WiFi somewhere where it could be anybody or install what is effectively malware on a machine somewhere to do it for you, both of these ideally with a program on your local machine that immediately bounces the data over TLS or perhaps even Tor to a server you've hidden somewhere ready to receive the data.
Basically, if you have to ask it's well beyond your skillset.
>>
>>60872887
>>60872962
Probably. I have someone else's credentials and a staff member. Obtaining multiple people's credentials to spread it out over a year wouldn't be a problem. Getting the data safely to a Tor server would be the part I need to learn.
>>
>>60873766
I just hate the fact all these science/natural philosophy databases are funded by taxpayers yet are closed off and so expensive. Even then, this is knowledge of humankind and they make a profit from it by keeping everyone ignorant who cannot pay?
I am surprised these databases have not been leaked multiple times before as nearly every university has access to multiple ones.
Thread posts: 5
Thread images: 1
Thread images: 1