How the fuck does Youtube manage to store everything when several

>>59897115
Yes.

>>59897115

>How much storage does Google have

A lot

https://www.youtube.com/watch?v=NTMkc0bLRlI

Holographic storage.
They have a mass drive of random sequences. When something is uploaded they make a matching pattern algorithm that corresponds to the sequences.
It's faster than reducing the raw data to a mathematical sum.

>>59897115
Youtube is a money, manpower and workhours dump that is yet to yield money to Google.

I guess google uses it as test ground for their big data algorithms? I have no clue, but if google ever gets in trouble, it's the first thing to go.

>>59897204
except entropy is so fucked on youtube video that that makes _NO_ sense. what patterns are they matching? all I, P, and B frames are gonna be compressed completely differently. pattern matching in a single video is impossible, but for the entirety of youtube? you're joking.

>>59897115
You know that massive NSA data center that Ed Snowden talked about and was in Citizen Four? Yeah, well all your shit goes there.

>How much storage does Google/Youtube have in total in 2017?
More than 3GB

>Using this method, they determined that Google holds somewhere around 10-15 exabytes of data. If you are in the majority of the population that doesn’t know what an exabyte is, no worries. An exabyte equals 1 million terabytes, a figure that may be a bit easier to relate to. Using our estimate earlier of a personal computer holding around 500 GB, that would mean 1 exabyte would equal 2 million personal computers, and Google's 15 exabytes would be around 30 million personal computers!

https://www.cirrusinsight.com/blog/much-data-google-store

>>59897115
>How much storage does Google/Youtube have in total in 2017?
About tree fiddy

>>59898207
But 500GB has been the standard for hard drive sizes for prebuilt desktops for close to a decade now.

File: storagecrystalsatgoogle.jpg (303KB, 1024x768px) Image search: [Google]

303KB, 1024x768px

Google here, we use laser inscribing on crystals.

>>59898207
An exabyte is 1000 petabytes
A petabyte is 1000 terabytes

>>59898488
Yes?

>>59898514
hes saying 1000 terabytes != 1000000 terabytes

>>59898543
1000*1000 = 1M

What's your point

>>59898543
Him and you either lack reading comprehension or don't understand basic math. Please leave this board.

>>59898460
can we take the botnet down with kryptonite?

>>59898564
>>59898583
D'OH

>>59897115
they use winrar (but its registered)

>>59898636
>I-I was just pretending..

>>59898488
>>59898543
>>59898564


That's a funny way to type 1024.

>>59898665
where did you get that? i see my mistake now, that was a simpsons reference

>>59898680
KB vs. KiB

>>59898376
there's no real need for more.

most normies don't even "produce" 100gb.

>>59898711

KB has been 1024 bytes since forever.

>>59897115
Could some malicious group upload huge amounts of trash to stretch Google's storage capacity to its limits?

>>59898751
Only in contexts where the amounts are naturally powers of two, so mostly bus-related stuff like RAM.

A 4 TB hard drive is most likely very close to 4,000,000,000,000 bytes, and a 1 Gb/s Ethernet connection really does transfer 1,000,000,000 bits/s, because there's literally no reason to measure those quantities in powers of two. That's why it really doesn't make sense to use binary prefixes for file storage or network transfer speeds.

>>59897245

They can match copyrighted work to every video, can't they?

>>59898680
>>59898751
Nobody was talking about binary.

>>59898751
that's KiB.

>>59898867

It's 1024 when talking about storage, not just in binary context. For data transfer (i.e. network speeds), it's indeed 1000. Check JEDEC.

>>59899017

https://en.wikipedia.org/wiki/JEDEC_memory_standards#Unit_prefixes_for_semiconductor_storage_capacity

Decimal only for serial communication data rates.

>>59898867

This is because marketing a 3.6 TB drive as 4 TB is better for the manufacturer than selling an actual 4 TB drive, not because a TB is anything less than 2^40 bytes.

File: 4L_DlTS1oFY.png (2MB, 1242x2208px) Image search: [Google]

2MB, 1242x2208px

>>59897271
well, you're not wrong.

>>59897148
/thread

>>59897115
I thought the same thing and someone pointed something out. The thing that help them out loads is de-duplication. I expect it is very common that the same info is uploaded on a regular basis.

>>59898929
That's looking for things that fit a signature. It's a different concept. It's why false positives exist.

Yes, and it costs them way too much money. Youtube's popularity and belief that you can "store videos on the cloud" for free is going to ruin it. Youtube should (and hopefully will) start charging content creators for operating costs, and leave them to monetize their channels on their own.

>>59897115
serious answer: shingled hard drives

It's kind of funny to me how little of specifics we have about google's datacenters

>>59899830
Facebook transitioned from hard drives to bluray discs, Google probably have something similar.

>>59899878
It irks me how little we know. Does Google have like a few huge, half empty buildings that they just keep adding to? Or do they buy new buildings when they need to add more servers and fill them up all the way at the begining?

>>59900002
there's a little we know there. They have buildings with full copies of all their (probably only public) data everywhere, to prevent lag. From some YT explanation or something.

File: 1491592519338.jpg (104KB, 410x271px) Image search: [Google]

104KB, 410x271px

>>59897245
>_NO_
Get the fuck out of here, Redditor

>>59898799
>group upload huge amounts of trash
You mean youtube users? Because that's what youtube is now. A bunch of people uploading trash in huge amounts. Taking down google by sending them lots of data is basically taking down the internet. They seriously operate on another level.

>>59900077
without google people would still have a few other sites, and those sites would just have to put up hyperlink sections at the bottom again. Like the good old days.

>>59899943
I'd say bait but they're probably cheaper than tapes.

>>59900064
How do you know that's from Reddit if you aren't a redditor

>>59900189
Common boding code applicable to places not here.

>>59900100
It's not an issue of whether or not the internet can survive without google. It's an issue of size. To bring down google with sheer volume of data would require a similar amount of bandwidth to bring down the entire internet. Maybe you could do it if you gained control of Netflix's computers and started scripts to upload movies to Youtube.

On second thought someone get on this. The copyright merchants would have a heart attack.

>>59900213
But how reddit specifically?

So what the fuck did the answer turn out to be here?

File: 1255485475541.jpg (77KB, 600x450px) Image search: [Google]

77KB, 600x450px

>>59900282
So not underestimate our autism for spotting things that don't belong.
Only a newfag redditor would do such a thing even tho they've never seen bolded text on this board.
Now go back to lurking, faggot.

>>59900350
I've been here for 5 years. I do not understand how underscores are a "reddit" thing. Do you lurk reddit for hours a day to be able to recognize their memes? What do underscores have to do with bolded text?

>>59900340
Not hard drives. All of youtube is stored on mercury delay toroids that encircle the globe in orbit.

>>59900370
I've been here for 10 years, and admittedly reddit for 5 years, and I have no fucking idea what underscores has to do with reddit. Granted, I've also never seen that used anywhere to begin with.

They already have all the data via library of Babel type design, and when you upload a matching video they just publish the already generated data

>>59900002
They're constantly expanding all over the world, they've just begun building their largest datacenter yet in Quebec, Canada.

File: le drink froge.gif (26KB, 273x200px) Image search: [Google]

26KB, 273x200px

>>59900064

> Sees common markdown code
> Instantly assumes Reddit

Stack Overflow and Github and damn near everyone who uses markdown uses the same basic syntax.

The fact that you don't know that and assume Reddit makes me think you're a Redditor... and nothing else.

pic related. reddit likes this meme.

>>59900133
You don't need to say anything, just google it.

File: 1481140613906.jpg (78KB, 600x450px) Image search: [Google]

78KB, 600x450px

>>59900350
I agree with you.

>>59900370
We know when redditors post here. You give off a distinctive stench like old fish and rotten garbage. Get the fuck out, filthy subhuman.

>>59900447
No one would be here since 2007, then turn around and openly admit that they use reddit. You've outed yourself and you have to go back.

>>59901874
>Stack Overflow and Github
>not redditors
Pick one.

>>59902334

> Pick one.
Wait, what?

> No one would be here since 2007, then turn around and openly admit that they use reddit.

That p.much describes me. Maybe earlier. I'm getting very oldfig and memory's the second thing to go.
Reddit's fine though. If you just eat one flavor of autism all day, it's nice to try a different one.

I admire your devotion though.

File: anon5.png (5KB, 225x225px) Image search: [Google]

5KB, 225x225px

>>59902388
>p.much
>I'm getting very oldfig
Are you 14 years old? Fucking kill yourself, you little shit stain. You sound like you've been here for a month.

File: fatorvalds.jpg (18KB, 500x316px) Image search: [Google]

18KB, 500x316px

>>59900350
Is he reddit, /g/?

>>59902408

The people who talk like you are usually the ones who are new.

I made you a cute emoji so we can be friends. ( ꒳ )

>>59902334
The _underscore_ to emphasise has been around much longer than reddit. It was more common before you even found 4chan (which was like what, 2013?).
It's always been taboo here though since it's retarded garbage, but since you call it 'reddit', you're the one who needs to go back.
What even is reddit anymore? I have no fucking idea since I never go there.

>>59898584
The botnet IS cryptonight

File: anon11.gif (233KB, 300x400px) Image search: [Google]

233KB, 300x400px

>>59902443
The ones who try to deflect or reverse accusations are usually underage faggots with an inflated sense of self worth who need to drink bleach and die.

You have to go back.

https://www.youtube.com/watch?v=XZmGGAbHqa0

File: plebbit.gif (925KB, 700x478px) Image search: [Google]

925KB, 700x478px

>>59902460
>defending cancer
Go back. Right now, nigger. Your kind is not welcome here.

File: reddit_front_page.jpg (341KB, 2448x3264px) Image search: [Google]

341KB, 2448x3264px

>>59902478

Oh dear. It seems you did not like my emoji.

I think we can still be friends though. I went to reddit like you asked and brought you back something from the home page.

I hope you like it.

File: 1f1xy6.jpg (54KB, 991x902px) Image search: [Google]

54KB, 991x902px

>>59902503
All of your posts scream underage b&. You need to leave.

>>59902525

Well if you're not even going to try then neither am I.

File: 1491361943686.jpg (237KB, 637x636px) Image search: [Google]

237KB, 637x636px

>>59902561
Get the fuck out of here, faggot kid.

File: 1488860576487.jpg (212KB, 800x732px) Image search: [Google]

212KB, 800x732px

compression.

>>59902460
>since it's retarded garbage
why would it be?

File: selfie.jpg (97KB, 550x825px) Image search: [Google]

97KB, 550x825px

>>59902568
you caught me
pic related. it's me.

on that note, I think I'm done here. was hoping you'd be more fun. :/

File: 1469138369054.jpg (10KB, 199x257px) Image search: [Google]

10KB, 199x257px

Will some big IT company eventually start making data centers on the moon or in space

>>59902609

Maybe once people are living on the moon/in space. I can't imagine that you get good latency to the moon from earth.

Plus you'd have to put transmitters all over the moon so that it can transmit from any side that's facing us. Same goes for receivers all over the earth, depending on where the moon is. Then it'd be routed to its destination.

Fun idea to think about, but not very practical.

>>59899943
I wondered that, that's why older contents are way slower to load, I mean, picture you uploaded 5 years ago. Also all Facebook pictures are recompressed under 90KB and the video quality is always shit. I think that's why they avoided GIF for a long time and now they're basically looped h264. I understand what they do, Facebook is their primary source of income and they must keep the costs down.

What happens if one of their storage devices break with YouTube content on it? They probably have a pretty good back up system though.

>>59897115
They store it inside your mum because she's so big

File: a.png (4KB, 458x147px) Image search: [Google]

4KB, 458x147px

>>59899038
>>59899017
>semiconductor storage
RAM, in other words. That's pretty much what I said. See picture for how hard drives work.

>>59899066
What I'm trying to say is that hard drives aren't naturally bound to powers of two, unlike RAM, so there's no reason to think of hard drive terabytes as binary terabytes rather than decimal terabytes.

The SI prefixes are decimal in every other venue of use. The only reason an exception is made for some things, such as RAM, is because RAM naturally comes in powers of two, not because "it's computers, so we do binary for everything". There's a reason the prefixes are normally decimal, and so if there's no particular reason to make an exception, then it's better to stay with decimal.

>>59902654
>Plus you'd have to put transmitters all over the moon so that it can transmit from any side that's facing us
anon...

>>59899038
>https://en.wikipedia.org/wiki/JEDEC_memory_standards#Unit_prefixes_for_semiconductor_storage_capacity

Did you read your own link?

>The specification notes that these prefixes are included in the document only to reflect common usage. It refers to the IEEE/ASTM SI 10-1997 standard as stating, that "this practice frequently leads to confusion and is deprecated".

>>59902609
Not as long as there's a one second latency to the moon.

>>59898207
Yeah they're definitely not using any technology we know

>>59897271
Hopefully they're not running the whole thing on a single Seagate drive in that case.

>>59897115
They also have backups and backups of backups.

Also spread onto multiple datacentres to load faster

>>59897115
>How the fuck does Youtube manage to store everything
alien technology

>>59898658
underrated post

>>59903179

Damnit. :<

I know I'm late at the party but I remember back in a ~2000,around when divx(later on xvid etc) became widespread,
I ran into argument with my friend, it was(at the time) total wonder that 2h long movie could fit on 701,59mb(we all used outer circle for another shoehorned mb).
Let's leave quality/compression ratio on the side, shit was watchable, almost DVD quality.
So, he said, in 5-10 years you will see 2h long movies on 50mb(this is still the era of 576p tvs,playstation2 ,NTSC/pal etc),
I told him that shit is just physically impossible(we went into argument, got into a fight and we still don't talk), lo and behold
circa ~2008!?(who will member all that shit) all known YIFY releases 720p at 300mb, some other groups that specialize in maximal quality/size/compromise managed to encode actually reasonable flix in under a 99mb@1080p/23fps.
Now, that was !?5-10 years ago... Now we have x265, it still isn't as good as x264 but it has space to grow.
But in future As far as video data goes, I see some master file that is really small and on demand it can be recoded up to a !?8k,
it sounds as space bologni now but so did 300mb 720p 2h long flix back then.
>in b4YIFY meme, I used it just as example everyone here is familiar (yes,you are..)

>>59904558
But how does it handle 3D Video?

>>59904479
No, it is not, its actual 4chan quality post from before, problem is expectations fell so low, this post feels like it won a 1000 interwebs

>>59904597
3d video is just stereo picture, so x2

>>59898730
most new games on steam now take like 50-100GB per game

>>59897115

H.265
.
2
6
5

>>59904630
i was making a Silicon Valley funny.

>>59904606
Preach it.

>>59904638
>most
I doubt that.

File: 2dd.jpg (15KB, 300x300px) Image search: [Google]

15KB, 300x300px

>>59903179
holy shit
took me a while to notice

>>59904696
all the AAA games that normies play do

>>59902436

>_G_

FTFY

>>59898799

Western Digital sys admin here

yes, we upload hundreds of terabytes of unique videos to youtube everyday so we get more money when Google buys our enterprise datacenter hard drives.

The videos must be unique since they use deduplication, and we avoid single color backgrounds because h.265 is very efficient at compressing single color areas

File: GavinFromSiliconValley.jpg (38KB, 640x480px) Image search: [Google]

38KB, 640x480px

>>59904597

>>59904724
t. Webdriver Torso

File: denpak.png (433KB, 689x632px) Image search: [Google]

433KB, 689x632px

>>59904773

>>59904773
Consider the armadillo..

>>59904696
Download any big name game released in the past 3 years, they're all 50GB or more

I think one of the CoD games was like 90GB

>>59905462
and usually it's retarded "uncompressed" audio as if anyone ever asked for that

File: 1357517468946.jpg (97KB, 492x759px) Image search: [Google]

97KB, 492x759px

>>59904638
>>59904696
wish I had a date attached to this screencap

File: THECRYSTALSCARTER.jpg (39KB, 400x225px) Image search: [Google]

39KB, 400x225px

>>59898460
kree

>>59905596
I guess most of it is textures or models and devs just keep adding more every year

>>59898488
Yes.

>>59898680
Fuck your feet and gallons faggot

>>59898751
KB has been 1000 since the 1700s.

In a brief period from 1960 some retards rounded it up to 1024 until the IEC had to step in

Companies storing data at that scale are using distributed object stores like Ceph, GlusterFS, etc. It lets them distribute data between different physical hosts (and more importantly different backing arrays.) As an added bonus: if you geographically distribute the hosts you can serve data to users with lower latency, which is desirable for their CDNs.

If you want to see what type of hardware they're using look at Backblaze. They do lots of technical writeups on their blogs about their storage pods, number of drives they use, what kind of drives they're using, failure rates of drives, etc. They tend to use large swathes of commodity hardware.

>>59905596
well, the CD32 came out late 1993
microcosm cd32 was 1994
the disks he's referring to seem to be amiga 880k disks
storing >200M wouldn't have been too crazy a few years later, not to mention CD burners did become affordable a few years later as well
so i don't see this post being much newer than 1994

>>59905596
i had no idea people have been saying literally the same thing for so long

>>59905596
"games are too big to pirate!" is meaningless when talking about steam, because you're /expected/ to download them there anyway

>>59897163
FAKE N GAY

>>59897115
Four.

>>59905487
One part is audio but another part is retarded way console game Deva were learned to code, since gen before this had,what!? 512mb?? If even that, so whole bullshit philosophy was streaming from DVD/CD/bray/ on 64mb cache, Prince,repeat(point was to not waste processing power on decompression)
And since most PC AAA games are console ports...
Also, devs let go old habits really slowly.

>>59905981
dude, the gen before last (ps2/gc/xbox/dc) used either synthesized or compressed streams, using compression allows for longer lengths of audio in the same sized buffer, giving the optical drive more time to read other data, while streaming music
the generation before that (psx/ss) often used uncompressed cd-audio, however, at the cost of not being able to load other things while streaming music (limited to games where it was feasable to load the entirey of the levels' assets into ram at once, bar the music). if not cd-audio, they used synthesized audio
everything prior to those was synthesized, more or less

tl;dr, streaming from optical media, or having low memory aren't reasons not to compress audio

>>59905981
>>59906164
-- also, if audio decompression was really that big of a cpu drain (it isn't), then they would have implemented a hardware solution for it, like they did with video decoding
an mp3 decoder chip is probably cents out of china

>>59906202
speaking of hardware audio decoding, i suppose one could include the dedicated sound processes on older, pre-streaming-audio consoles (and home computers/arcades)
they're not decoding audio /streams/, but they are doing all the work getting the stored version of the audio (an executable for the sound chip), "converted" over to actual analog audio to be listened to

so really, an audio decoder wouldn't even be a new thing to consoles, just different

>>59897115
it was > 24 hours a minute years ago

File: 1437958505958.png (226KB, 620x670px) Image search: [Google]

226KB, 620x670px

>>59904724
>yes, we upload hundreds of terabytes of unique videos to youtube everyday so we get more money when Google buys our enterprise datacenter hard drives.
that's one idea for making money

>>59906202
Sorry,I'm phone posting, I meant chunks of game,not only audio.
My bad.
Point being, Skyrim is 1,7gb (original PC game full rip without foreign languages,lossless).
Now,normies couldn't believe such Hueg gaem fit onto 2gibs.
I bet fallout4(idk the size but I bet ~20-30gbs) could be fitted under 5gb easily when all crap is cut out,although they use some high red textures here and there, first time in history

>>59907073
what did he mean by this

>>59907157
>Skyrim
>lossless
skyrim's audio is compressed to the moon and back.

>>59902334
>we're very good at sperging out about boogeymen we know nothing about because it makes us feel like we fit in on this secret club
FTFY

>>59897115
They just keep buying more cheap flash drives off eBay and plugging them into USB hubs, so it's just a constant stream of flash drives from China to YouTube hq. Legend has it that over 50% of shipping containers on ships in the Pacific are full of flash drives shaped like monkeys and bananas and all that shit

>>59897271
That's correct.

Which filesystem does youtube use?

File: 1491894524112.jpg (87KB, 607x451px) Image search: [Google]

87KB, 607x451px

>>59903221

>mfw

>>59898460
>the Fortress of Solitude is made up of millions of these crystals
Just how much furry porn do you think Superman stores in there?

File: joyful stare.jpg (47KB, 500x503px) Image search: [Google]

47KB, 500x503px

>>59902489

>>59897115
Not only do they store your original video file, but also all the re-encodes of it. I can't even fathom how many hard drives get used up per hour on that site.