/dlg/ - Deep Learning General

>>60707174
Book for beginners:
http://neuralnetworksanddeeplearning.com/index.html

Book for intermediates:
http://www.deeplearningbook.org/

>take NN course
>still confused about the differing details of implimenting a neuron and a perceptron based system
>still havent actually implimented backtracking
>only have a vague understanding of how it works
i think ill have a go at knocking out a basic single threaded proper NN at some point soon, NNs are fun

>>60707248
>backtracking
i mean backpropagation, because apparently im retarded

>>60707248
Check chapter 2 of the beginner book I linked above

>>60707174
>Linear Algebra
Check

>Multivariable Calculus
How much? Just partial derivatives? I've done calc 1 so I could easily learn that.

>Statistics
Done an introductory statistics course, so check I guess.

>Python
LMAO
M
A
O

I was wondering why people were complaining about machine learning being slow, but now I fucking know why: They're using the slowest fucking language!

>>60707248
This is because NNiggers intentionally obscure what big data ML is, because VC money bubble.

It's just a fucking decision tree evolved by brute force instead of inferring the hidden variables in more inductive way a real human bean would.

The xor problem is "avoided" by chaining bunch of those (so called layers) to the point you enumerate all XOR corner cases (ie XOR like problems are implicit overfit).

Gradient and descent and backprop are to narrow the search space. Meaning very simple NNs (fe simple captcha breaking) really don't need one. Just like simple chess algorithms don't do any tree prunning, and just bruteforce and rank very few steps ahead.

Thank you for making this lovely thread anon instead of making a intel vs amd thread.

Pls teach the new memes in AI.

>>60707914
A rare moment of someone on /g/ knowing what they're talking about

>>60707860
Python is a good prototyping language. It's there to glue the efficient parts written in native C/C++/CUDA together for ease of use and with a well written input pipeline it should never be your bottleneck.

Most big DL libraries have (often exclusive) Python APIs, like Pytorch (Facebook), Caffe2 (Facebook), CNTK (M$), Tensorflow (Google) and Theano (Canucks)

>>60707174
>>60707183
Memes, thats all. Not a single working application has been released by any of the authors claiming to experts in fields like deep learning, ai etc.

>>60707860
>Done an introductory statistics course, so check I guess.
This generally isnt sufficient if you want a good idea about what's going on. Machine learning is basically applied statistics.

>Lolling over python
The bottleneck in ML is seldom processing the data and more storing/acquiring/transfering it in the first place. Also numpy uses BLAS which is bretty good for array operations anyway.

>>60707955
Demis Hassabis wrote the original RollerCoaster Tycoon.

>>60707955
This is like accusing mathematicians of not being engineers.

>>60708019
No its more like writing books how to be successful when the author is a poor loser.

Mathematical knowledge does not translate to deep learning or AI otherwise we would have had fully functional AI years ago.

>>60708096
lol Schmidhuber basically invented AI back in 1923

>>60707174
Just getting started?

>You need at least the following at introductory levels:
>Linear Algebra
Done
>Multivariable Calculus
Done
>Statistics
Done
>Python
Why? I use MATLAB

>>60708186
See
>>60707948
>>60707961
TL;DR: Everyone uses Python, and it's effectively the only one available.

>>60708096
>No its more like writing books how to be successful when the author is a poor loser.
Why?

Coming up with a very effective algorithm doesn't necessarily mean you have an effective use case for it yet.

Case in point: Bayesian Optimisation has been around since the 1970s but it's only recently found it's real niche due to the lack of computing power available in those days.

>>60707860
>How much? Just partial derivatives? I've done calc 1 so I could easily learn that.

PDEs are the most obvious and commonly used method to deal with minmax in mainstream frameworks, but there are drastically different approaches to ML too which are conceptually similiar in engineering terms, but somewhat different math.

The opposite to PDE on this spectrum would be binary NNs, where minmax is performed by a SAT solver (basically partial MQ polynomial in GF(2)).

General MQ polynomial math of all kinds - real, integer, field, groups - can be all used, because the only thing training algorithm need is "approximate future input trend for given output trend", and a lot of math subfields exhibit and deal with this property, not just real/complex field PDEs used in mainstream engineering.

>>60707174
You should probably add probability theory up there too, OP

File: b49.jpg (36KB, 680x680px) Image search: [Google]

36KB, 680x680px

> muh ReLU: reinventing LP constraints
> muh CNN-s: reinventing tunable filter banks
> muh RNN-s: reinventing tunable IIR filters

Thank you based Hinton zombies for reinventing optimization and signal processing in the third millenium.

>>60708186
You don't HAVE to use python but it will save you a lot of boilerplate.

>>60707961
>Machine learning is basically applied statistics.
I disagree. It appears to be more applied calculus to me.

>tfw should be learning AI in the next 1-2 years at uni

hopefully i can learn it early and well enough to become an early adopter

>>60708490
Look at the bright side, all this "NN-ready" hardware coming out of this fad can be used as pretty fine DSPs.

>>60708572
It's too late. Successful machine learning researchers are identified in elementary school machine learning competitions. Only the most creative, innovative, and gifted students are selected. If you were never aware of the process, then it means that you failed in the secret initial qualifiers, and weren't even close to earning a place in the program. This process may sound harsh, but it would simply be cruel to try to train someone in the art of machine learning if they don't possess the raw talent.

>>60708347
Are there any clear advantages of using other methods than PDEs?

File: whynot.jpg (27KB, 350x350px) Image search: [Google]

27KB, 350x350px

>>60708572
Why should humans bother/ Why not just teach AI how to tune ML models if it's too hard for bones and flesh? The model parameter selection is already black magic alchemy at this point.

I can't wait for firmware for misogynic killer robots either, anon.

>>60708636
No quasi-newton method or stochastic optimization method will measure up to minibatch SGD on all but toy problems like MNIST

>>60708675
there are already techniques that attempt to "learn" the correct parameters.

>>60708675
Large scale hyperparameter optimization is still pretty nascent, bayesian optimization mentioned in >>60708247 is one of them

>>60708675
Also when it takes a week to evaluate the function (loss on validation set vs your parameter) once, you bet human insight still beats automatic parameter selection.

>>60708636
There are tradeoffs. Disadvantage is that those approaches are much more difficult to reason about - how do you minmax in a chaotic ring?

There are ways, but not obvious. For example, you can define minmax not in terms of real function derivative, but in terms of population representation in a cellular automata, or as a group operator in chaotic group, or simply sets of on/off bits (binary weight NN).

Advantage is often much better performance (especially in case of binary NNs). There's also possibility to express chaotic states as partial statistical component of input/output pad sets which linear functions simply can't do.

If you want to see some PoC code out there, try https://github.com/allenai/XNOR-Net for example

>>60708554
I forgot I was in deep learning general actually, in which case I would be inclined to agree

File: traprogrammer.jpg (115KB, 1000x900px) Image search: [Google]

115KB, 1000x900px

Reminder that wearing your programming socks make you a better deep learning researcher.

>>60708906
Wait, what's the difference between "normal" machine learning and deep learning that makes deep learning more like calculus?

File: xaxa (2).png (25KB, 424x213px) Image search: [Google]

25KB, 424x213px

ML isn't a trap, as long it's cute.

>>60709011
Buzz term. But it refers to multi-layer networks, as opposed to simple single layer NNs.

>>60709046
>But it refers to multi-layer networks, as opposed to simple single layer NNs.
I know that, but how does it make it more like calculus?
Can't you apply gradient descent to single layer nets?

File: maxresdefault.jpg (191KB, 1920x1080px) Image search: [Google]

191KB, 1920x1080px

lmao why bother with all that crap when I can just learn electronics repair and resolder macbooks for mad cash 24/7

>>60709066
In single NN it's called delta-rule. GD is simply incremental generalization of it over layers. But in both cases you deal with approximating inputs for some outputs, making it "calculus", I guess.

>>60707257

Its simpler than what its made to be.
https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/

>>60708619
This is absolute BS

>>60709632
Then answer me this. Why were Demis Hassabis and CERN's Director General invited to Bilderberg meetings?

>>60707860
python is only there to control the general flow of the program.
most of the work in machine learning consists of doing calculations over huge matrices. Those operations are usually implemented as C programs which you can conveniently call from your python program.

Machine learning is slow because machine learning is complicated.

>>60708554
But machine learning is actually a field of statistics.

>>60709721
> Why are two people at the top of their field invited to a meeting for the people at the top of their respective fields.

You do not *need* to be a child prodigy to make significant contributions to machine learning or anything else for that matter. Of course people will be interested in the vision of the pioneering minds but to claim you'll never make a contribution because you're not Jürgen Schmidhuber is ridiculous.

Yes, there are people who fall into this category, and yes there are sudden paradigm shifts occasionally, but the vast majority of scientific and/or mathematic advancement is one tiny enhancement or observation at a time, built out of a network of contributions and feedback from the community.

>>60710112
For people who want to start yapping about the last group: All those problems are obviously intractable. The joke is that advanced AI capable of "reasoning" about such problems is intractable too.

Or humans truly are too dumb to deal with those, and intractable simply reflects our ignorance and limited capacity.

>>60708619
Le reddit ML troll

File: never everr.png (568KB, 600x580px) Image search: [Google]

568KB, 600x580px

>>60707174
>Linear Algebra
>Multivariable Calculus
>Statistics
>Python

L O L
O
L

How to maching """LEARNING""":

import sklearn

>>60710112
>made by a highschooler

>>60708207
>what is R

>>60709066
autodiff

>>60710112
> one time pad decryption

File: .jpg (59KB, 600x637px) Image search: [Google]

59KB, 600x637px

>suddenly realize neural networks are meme trash and solve my problem using actual statistics in 1/1000th the resources and 1/100th the development time

its an enlightened feel
im better than all of you

>>60707914
i mean i get that much, ive done a module on general machine learning using weka before, its just i have an incomplete understanding of the actual process of backpropagation and weight sliding. for my project i essentially resorted to just individually sliding the weights based on T-O delta * learning rate, which yielded better results but clearly isnt how things are supposed to be done.

>>60707279
ill read it later senpai, thanks

>>60709616
i just need to get round to implimenting it, but its a bit late in the day now

>>60707174
>Linear Algebra
>Multivariable Calculus
>Statistics
>Python

That doesn't seem too bad

>>60714039
>to just individually sliding the weights based on T-O delta * learning rate, which yielded better results but clearly isnt how things are supposed to be done.

This is pretty close approximation of delta rule you do for a simple NN.

>>60713235
Not quite. The power of ML still is that it is fairly general and "fuzzy", while more powerful algorithms, symbolic logic, PCA, SVM ... work only on well defined problems.

You're right that a lot of ML tasks are now thrown at ANNs simply because it's cheap to do so in terms of development cost, and hardware cost is neglible in comparison. It's time to market bloat like any other - often, ANN is to expert system like electron is to desktop apllications.

File: 9PVle5o.jpg (283KB, 900x1200px) Image search: [Google]

283KB, 900x1200px

I'm interested in neural networks so I read this karpathy.github.io/neuralnets/ and implemented a basic network to classify iris species(https://archive.ics.uci.edu/ml/datasets/iris) so far. I want to implement my own library for neural networks but I haven't found any good info on implementation architectures/types of implementations. Any good resources(or good code to study)?

>>60714213
yeah, but its not really the right way to do it though

>>60709119
The mark of the true pajeet. Some of us are in this scientific field because of genuine interest.

>>60714309
For starting toys, I'd recommend:
https://github.com/NathanEpstein/Pavlov.js - reinforce/markov decision trees, not strictly NN
https://github.com/karpathy/convnetjs - small self contained CNN

And to get insight how heavy duty frameworks tick, read:

https://github.com/BVLC/caffe

caffe is a "vanilla" framework. it doesn't try to scale to bazillion gpus, or have a nice end user UX and whatnot, instead, it has a very compact, readable implementation and documentation, so one can get idea how things are actually done -how all the solvers and filters are put together, as well as implementation of each module.

>>60714682
Thanks

>>60707174
>try to implement CNNs myself
>fprop: fucking easy
>bprop: should be easy too
>wait a minute, no math I've read covers any nontrivial case (padding, stride)
>finally find a normal paper on it
>try to look at easy to read libraries -- they're all fucking wrong in their implementation of backprop
>end up copying Caffe's approach
Like and subscribe

>>60707174
what are some good beginner level projects for machine learning?

>>60707914
wow someone who gets it on /g/