Is compression at the limit? Seems like there hasn't been

It requires increasingly complex statistics to push compression further in a manner that's still practical. The other thing you're talking about isn't so much a matter of compression, though. That comes from us rubbing up against the laws of physics. We can only make the physical structures which store data so small before they lose fidelity over the long term. To push further we need to use different materials and methods.

Dude just use middle out compression

>>61477532
Further compression of storage methods gets horribly expensive fast.
This is why HVD died off pretty quickly. The DISCS alone were 5-digits!

Compression of data, likewise, gets horribly CPU-intense the harder you try and push it.
We're getting pretty close to statistical compressions limits.
Beyond that, it is hard brute-force compression which takes considerably longer.
We're literally speaking "every file is in Pi" method. (see PiFS)
https://github.com/philipl/pifs
A useful tiny file in PiFS would be well up in the millions of iterations.

>>61477532
>Weren't we supposed to be storing petabytes of data on crystals the size of a pin head by now?
Maybe corporations, but that tech is a long ways of from being in consumers hands.

More importantly, magnetic tape is still the preferred way of storing large amounts of data.
Like a 185tb LTO-6 tape is only about $60, not including the drive to read it.

Lasering data into a crystal on the other hand needs a electron microscope to read and write to it, and the crystal is pretty expensive since it has to be flawless.

https://en.m.wikipedia.org/wiki/Jan_Sloot

?

>>61477532
>Seems like there hasn't been any advancement at all for the last decade.

RAR5 surpassed 7z like 5 years ago now (it has better compression while also compressing twice as fast).

Video formats are getting new compression formats practically yearly nowadays. VP8, VP9, V10, H265, AV1, plus all the 10-bit formats for each. Also support for 4:4:4 and 60fps.

Audio hasn't seen much advancement, only in low-bitrate lossy environments, which don't matter.

So yeah, there were plenty of advancements in the last decade, you are just living in a cave.

>>61478679
also, there were new image compressions like WebP which compress better than JPEG.

Lossless compression is pretty near its limit.

Lossy compression however isn't.

>>61478706
Lossless is infinity more useful though.

>>61478679
>low-bitrate lossy environments, which don't matter.

dum nig, that's the only thing that does matter. Opus is the future.

>>61478679
>RAR5 surpassed 7z like 5 years ago now (it has better compression while also compressing twice as fast)
proof?

>>61477532
Well compression is still making progress for specialized data. Like brotli algorithm for http. It uses a small 120kB dictionary but that improves conpressability by a lot.
HEVC aka H.265 and opus are pretty fucking impressive. 3 mins of 320 mp3 have about 7MB. You can now have 3 mins of acceptable quality low res video+audio at 7MB.

>>61477532
>Seems like there hasn't been any advancement at all for the last decade.
Plenty of work happened for multimedia content. It's just too patented to be worth looking at.

>>61478513
So from an information theoretic standpoint what exactly is the compression limit using the π method? I assume the index numbers aren't going to be very small even if you use smaller chunks of data.

>>61477532
Instead of making compresion more powerful, just give us 10tb cheap ssd and high connection speeds.

ye

HEVC and Opus is about as compressed as video/images and sound can get.

Everything else LZMA2 is about as good as it gets.

We need 1TB SSDs and 256GB microSD cards to start dropping in price asap.

>>61480323
Not 100% sure, but probably similar in how the file table of a filesystem represents the file locations.
A typical FAT is considerably smaller than the actual partitions contents.
Given PiFS is designed to be used with split file chunks, it would be even similar in that regard.
Using it for one single continuous file would be insane. So, your files will be fragmented by necessity. (not that it would be an issue since this would purely be for archival reasons)

At present, doing stuff like this isn't really wise since there isn't hardware around to accelerate such operations.
There's been no push for it.
Generic hardware acceleration just won't work as well for it.
Not to mention last I checked it doesn't have support for it. (PiFS that is)
Another alternative would, instead of brute-forcing it constantly, would be literally caching regular sections of code used in files.
If it is ever found in more than a certain threshold, cache it. (say, 20 files)
A common example would be caching the header information of file types, API calls, etc.

Fun to experiment with, though.
Another fun file system experiment was the OFFS - owner free filesystem. Dev probably got suicide'd though.
Mind you, IPFS seems to be taking off as well as numerous P2P storage systems.

>>61477532
the limits of compression have been known since the 50s, the focus is currently on speed/parallelism with xz/lz4/lzo