New compression protocol?

[jarlmaster475]

As games get larger and larger, it feels like something has to be done to combat this. Can't we further compress files? Is there some new compression protocol or technology that we can use to reduce game file size? It seems to me that there should be some room for innovation and development in these basic file system technologies.

 

[beers]

Most games already implement compression on their resource files. It'd be cool if you could magically turn 4 petabytes into 1 KB (I guess you could with a data set consisting of all zeroes or all ones), however what do you define as the realistic limit? Do you have any knowledge or development of your own compression algorithm, or what specifically do you feel that algorithms like LZMA2 are lacking?

 

[jarlmaster475]

I don't know anything about compression algorithms lol hence why I was asking. I'm curious about it. My philosophy is that, within limits, everything can always be better. More efficient, faster, more accurate, etc. I know that games do implement compression but file sizes have ballooned in the last few years. I understand why (more complex games, higher res textures, etc.) but I wonder if there's room to improve on said compression or not. Perhaps there's a limit in compression that we've reached and can't surpass until quantum computing becomes mainstream or something. Or perhaps it's that no one wants to put in the work. I don't know. If someone could elucidate me I'd appreciate it.

 

[Darren]

Reason games have gotten noticeably bigger is high resolution textures which can't be very easily compressed. You need to have a pretty good understanding of data structures and programming in general before you can even really think about making a new compression algorithm that hasn't already been done. There's always a way to improve stuff sure but you're kind of asking why we haven't cured cancer yet, stuff's complicated the deeper you get into it and not something you can really have readily answered on a forum like this that isn't armed with data compression scientists

 

[jarlmaster475]

Fair enough haha. As far as cancer goes, we've made impressive strides. That's what I'm asking for with compression and file size. I don't need a "cure" per se. Just improvement and progress. Something in response to this trend of game file size getting bigger and bigger. It's somewhat akin to the increase in ram usage/density leading to the necessity of a 64 bit operating system. And I wasn't expecting an in-depth response nor did I want one lol.

 

[beers]

No worries @jarlmaster475 I was just trying to get a bearing on how/where/why your expectations are set they way they were.

As per @Darren a lot of resources contain incompressible or already compressed data which you can't 'double compress'. Certain algorithms are better suited for different tasks as well. Algorithms like LZ4 sacrifice a bit of compression efficiency to be exponentially faster at compression/decompression in comparison to something like LZMA2. With commodity hard drives being relatively inexpensive you'll find that most will go for faster algorithms (ie, shorter load times) over saving a couple of megabytes.