>I fell for the 16GB of RAM meme should've gone for so

>>59848385
Well, among the obvious options you could try optimizing it or writing it in a more efficient language. What kind of calculations does your program do that you run out of memory though?

>>59848438
training a machine learning algorithm with a 2.3GB dataset

>>59848385
I love 16 GB of RAM.

File: htop.png (54KB, 1366x745px) Image search: [Google]

54KB, 1366x745px

>>59848385
> he fell for the “8 threads is enough” meme

>>59848385
>>59848442
Optimize it or throw it in the trash. This is unacceptable.

>>59848438
>>59848508
>two people telling me to optimize
how tho

>>59848438
>>59848508
> just optimise this machine learning algorithm
bruh, knowing how to change and improve it is a big step up from just knowing how to use it
Not OP, just someone who ragrets going into ML for postgrad

>>59848442
retard

did you even try to optimize the dataset? if you have such amount of data you get fuzzy output. why not just feed rng into the maschine. wont make a difference on the output.

>>59848574
I did. I'm only using three out of >30 columns, so it's not the entire 2.3GB.

>>59848442
>>59848598

>ITT
>OP teaches maschine learning algorithm how to be a faggot by feeding it 2.3GB/30 rainbow flag jpeg

>>59848617
actually I'm teaching it how to predict the profile of a murderer based on his victim and weapon of choice plus a few other things

doubt there's much correlation anyway but I'm bored and have nothing else to do

>>59848617
How is it supposed to be useful when you only give it a fraction of the actual data?

>>59848659
do you want to make the maschine a full faggot? we already have OP

>>59848655
if you compile a dataset with jut such stats it cant be several GB even if you take the whole population of a country and a 100 year time scale

how are your data formated? are they even formated?

Stop using shit written in Python. That language has the most worthless memory management and garbage collection on the planet, which is just another reason why it's the worst language ever created.

>>59848659
I'm still in the preliminary stages of working with this data, and seeing as how I can't even get this small subset to work I may have to reconsider this project

>>59848679
>>59848617
>maschine

>>59848704
>quasi-English sentence which I think is saying my dataset can't be that big
lies
https://catalog.data.gov/dataset/crimes-2001-to-present-398a4

>>59848715
recommend me some ML libraries in C++ that are as simple to use as Python's and then you'll have a decent point maybe

>>59848385
>using python
>wondering why no memory

File: file.png (21KB, 677x136px) Image search: [Google]

21KB, 677x136px

>>59848465
Shit I only have 16 threads. Do I need to upgrade?

>>59848741
>Using python for machine learning
>Doesn't understand why he has no memory free

>>59848741
>I want to do machine learning but I don't want to write anything myself
You're worse than a web developer

>>59848762
You add more CPUs; I'll add more RAM.
Agreed?

>>59848777
I've been doing machine learning for 2 days so I don't really know what I'm doing

File: giphy.gif (488KB, 500x200px) Image search: [Google]

488KB, 500x200px

>>59848741
>quasi-English sentence which I think is saying
>lies

just because they offer these data to you doesnt mean that you have the ressources to handle them. you must be special if you cant come up with that yourself.

also dont act like you know what your doing
you try to take an algo from the web and some data and think you can just somehow blend them together and get a usefull output.

>>59848848
your first point is completely irrelevant

your second point is completely correct; I have no idea what I'm doing and don't claim to

>>59848508
optimise a library?

>>59848811
Then you should especially not be using python, you should be learning how to build simple neural networks and develop feedback systems.

If you were a data scientist with access to a server cluster to just toss some python scripts on to run for 5 days with no concern for resource consumption, that'd be fine.
But that's obviously not the case, so do it the right way and learn something.

File: lonk.png (31KB, 243x289px) Image search: [Google]

31KB, 243x289px

>>59848385

sudo ulimit -m unlimited

>>59848777
im a data scientist and everyone i know professionally in the same field uses either python or R which are high level languages

>>59848886
yes, it open source isn't it? get to work freetard.

>>59848867
here's a tip
learn about the algo you want to use and what works best with it.
then learn about your dataset and what pieces of the data are subsets you can use to get a usefull progress ratio out of your time invested. train your algo and the chunks and see if you get the result that you expect. if not you failed and need to recalibrate.

its not a magic box you need a target you aim towards or you cant interpret the data in any frame and that means you got gibbirish

>>59848934
only fucking academics, open source contributors, and phd students write original machine learning code in low level languages

File: 0.jpg (101KB, 430x637px) Image search: [Google]

101KB, 430x637px

>>59848927
>having a job
>having a non-autistic view of higher-level languages

>>59848927
see
>>59848903

Python is acceptable to use in some situations, but they are very far and few between

>>59848442
>Noobs blindy pushing retarded amounts of data in to a general purpose python neural network library
What else is new.

>>59849025
holy shit have you ever used scikit learn? do you know what a library is?

>>59848385

File: 1410294462409.jpg (88KB, 724x720px) Image search: [Google]

88KB, 724x720px

>>59849036
> thinking a random forest is a neural network
hehe

>>59848919
viruse

>>59849230
not really

>>59849230
wiruss

File: mpv-shot0019.png (1MB, 1920x1080px) Image search: [Google]

1MB, 1920x1080px

My friend, look into this sysctl:
vm.min_free_kbytes

And friendly reminder: Linux sucks balls. BSD slightly better. seL4 is where it's at.

Create a swapfile

>>59848385
>i3
>muh key combinations
kek

>>59848385
I want to post my desktop but desktop threads are banned, the thread.

>>59848533
Look up lazy evaluation, and how to write generators in Python. You will want to process the data set in manageable chunks of memory. I.e load 300 mb of data into memory, run your linear classifier or regression heuristic, save the weights matrix or whatever results, release that memory and then load in the next chunk of data into memory.

>>59848385
Jesus christ, do you wear a manbun too?

>>59848442
>machine learning
fucking tryhard memester

>not having an 18c/36t xenon CPU with 64GB of RAM

>>59848533
Feature extraction and dimensionality reduction.

>>59850187
>>59850187
This, buy an ssd and create a massive swap file - no regrets

>>59848385
I bet your shitty program is relying on a swap you don't have.