File Compression Techiques/ Methods.

[Peon]

I plan to compress all my games, programs, patches,tricks and anything else onto my file server.

What is the best program to use or is there is a certain specific way like splitting the files into archives that might work better.

Thanks.

[Pyro]

7-Zip seems to get the best compression these days ...

I don't get what you mean by the rest.

[<A>Nom@n]

hmm i think he means he needs a backup program and organiser... not just compression proggie.

[Peon]

Naw Noman. I just need something that will compress stuff very tightly :P


Im just asking what people reckon is the best program or tool i could use.

Hmm, pyro i will try 7-zip.

Thanks.

[Shrimp]

Hmm, I found 7-zip to be really painfully slow. If you're backing up a lot of stuff as it seems you are, you're going to sit for a good couple of very long hours waiting for it to do it's thing.

I'm not sure how fast or slow the ACE format is, but it does compress very well - look for WinACE and give that a try. There's also RAR which is slower than ZIP (plain zip, not 7-zip), but you get much better compression. Not as compressed as ACE though. Check out WinRAR for that.

[<A>Nom@n]

what i like about winace is it compresses better than rar or zip, and it has a lot of very nice advanced things you can do with the compressed files "out of the box", like perform commands after extracting the file, run a file after extraction (i.e. setup.exe) etch. oh and it can extract and compress the other formats as well...

[hai2u]

Most of that stuff is probably uncompresable, I really wouldnt expect you to get a ratio of more than like 10000:9999 or something not worth the CPU time - just buy another drive, compressing is a waste of time and a nucience.

[Peon]

WinAce looks promising aswell. Ive been dabbling with rar a bit. Its ok but doesnt do the best job i think.

Will fiddle and post results, sometime this w.e

Thanks guys.

[Pyro]

Ace was king for a long time, but rar has overtaken it again last i checked. 7-Zip is the best though, and if you have the memory for compressing at larger dictionary sizes (32Mb dictionary needs about 384mb FREE) it has great ratios. Yes it's not so fast when you push it to this level, but decompression is still fast, and only need your dictionary size (32mb) + a little overhead.

If you compress with lower dictionary sizes, the compression+speed is comparable to RAR at same dictionary sizes (8mb iirc), though haven't tested it in a while.

Haven't bothered much recently, since most stuff I have is already compressed in some way, or on CD/DVD's (eg Mp3 + mpeg4 movies). Zip suffices nicely, and 7zip does for when i really need to squash something onto my flash drive. I'm also not gonna pay the money for legal Ace+Rar so ...

[Peon]

Hmmm


7-zip is a crock of shit. The same archive it wants to make 1.13 Gb from 1.55 Gb as an example.

WinAce is kewl and all but does the same job as WinRar even when you fiddle with settings and things. However WinRar is much easier than WinAce and i think a bit quicker.

So I reckon WinRar should be the way to go.

[Terror]

Thats what i use, if u want Corporate editon gimme PM angel.

[Peon]

That would be cool, would appreciate it.


Thanks.

[Terror]

enjoy :DD:D

[Pyro]

Rar better than 7zip? Why?

Tested these just now using some UT2004 related stuffs
Rar - 3.42
7Zip - 4.20

Custom profile was simply Fast profile with a 4mb dictionary, not very effective, but closer to rar in patch test.

This would keep you happy with RAR, speedwise:
UT 2004 - 3355 patch

36840k - Raw
12952k - Original
11288k - 0:12 - 7z - Fast
10566k - 0:34 - 7z - Custom
10588k - 0:32 - RAR - Best
9417k - 0:54 - 7Z - Normal
9012k - 1:50 - 7z - Max
8893k - 1:51 - 7z - Ultra


But, this is way more in 7zip's favor
UT 2004 - ECE Bonus Pack

207370k - RAW
86457k - Original
76400k - 1:17 - 7Z - Fast
64587k - 4:23 - 7Z - Custom
59009k - 4:52 - 7Z - Normal
67258k - 5:47 - RAR - Best
57762k - 7:57 - 7Z - Max
Ultra - will have to wait for another time...


NOTE: These tests weren't under exactly controlled conditions, I was busy downloading over my wanna-b-56k, and typing this in notepad... Timing was also according to time elapsed in window, not atomic clock.

Like I said, if size is what you are working for, 7Zip is your answer, RAR is good, but not that good. If you want a snazzy GUI, try power archiver, not free, but neither is WinRar. Running it at maxed out settings is madness though, unless you have some kind of supercomputer.

[Pyro]

Another comparison:

eclipse-SDK-3.0.2-win32.zip

93869k - Raw (contains plenty of compressed content)
87491k - Zip (Original)

81174k - 3:35 - RAR - Fast
81065k - 3:54 - RAR - Normal
81040k - 4:25 - RAR - Best

82839k - 0:55 - 7Z - Fast (with 7-Zip's front end)
80870k - 3:28 - 7Z - Normal (with Power Archiver 9.20.07, probably old though)
80873k - 2:51 - 7Z - Normal (with 7-Zip's front end)


Maybe my old AMD XP 2000+ is running a bit weird with 640mb of DDR @ 333, but so far it seems 7Zip is faster AND better.

[sneaky]

Been using RAR for a couple of years now, happy with it. So dont see any reason to switch.

[Shrimp]

Your system must be smoking crack or something Pyro :P.

I just compressed a 355MB Interbase database file, using 7-zip, with a 4MB dictionary as you said you did in your test, and only on Normal compression.

That took almost 10 minutes, resulting in a 9% compression ratio (33MB).

The same file, using WinRAR, best compression, took exactly 4 minutes, with a 11% ratio (40mb).


I can't see the benifit of waiting more than double as long, for a file that's 2% smaller. I hate to imagine how long it would take on the highest compression mode.


I guess your tests were different since you're zipping already-compressed files. As far as I can see, 7-zip is like a super-optimised version of the zip format, so it's probably capable of making use of the compressed data already there. RAR on the other hand is a totally different format, so it can't make use of the compressed information.

Normally I don't zip my zip files anyway :P.


Aaaanyway, I guess it's personal preference at the end of the day. My preference is to spend half as much time compressing, and living with a file that's 2% bigger than it could be :P

[sneaky]

So, you will zip then RAR then 7-Zip to get the maximum compression :P strange thing is people actually do this.

[Zyron]

Most of that stuff is probably uncompresable, I really wouldnt expect you to get a ratio of more than like 10000:9999 or something not worth the CPU time - just buy another drive, compressing is a waste of time and a nucience.

Karnaugh's right you know.

[Shrimp]

Quite. However we are debating compression, not disk space availability :P

[<A>BioGizzard]

well i've been using rar from since i can remeber and i like it so i don't c the point in getting a diffrent format when the one i'm using works fine for me

and i don't care how long it takes or how much it zips more or less that the others

so imo shrimp is right....personal preference

[Pyro]

I weren't compressing pre-compressed files (except for the Eclipse test that is). If you keep compressing compressed files, you'll quickly reach a point where the compressed file is bigger than the files being compressed (Entropy I believe it's called). On the UT tests, Original is in .EXE as I downloaded them, RAW is the uncompressed size before feeding it to rar + 7z.

I used the FAST profile with a 4mb dictionary, NORMAL would take significantly longer.
(fast uses fast LZMA, and HC3 Match finding, normal uses normal LZMA with BT4 matching).

If you want speed put it on fast.
If you are looking for a bit more, go normal.
If you want high compression use max.
If you can let it run through the night, put it on ultra.

Don't fiddle with the specific settings unless you want something specific, they are pretty well thought out, eg, there's no point in searching for large words if you only have a tiny dictionary, and a tiny dictionary won't be helped by a complex search/matching pattern. I was just putting it on 4mb dictionary to give results comparable to RAR.

Lastly, for most tasks compression is pointless, but there are uses. If you're looking at downloading up to your 3G telkom cap, which would probably be locally capped as well one of these days, that 2% is quite a bit... I compressed this entire UT2004 folder I have here. Down to 2.60 Gig with 7Zip, down to 2.83 with Rar. Would be nice to have 200mb extra on that cap, wouldn't it?

If you were to host the UT 3355patch, you'd save 4mb of bandwidth for every user downloading it. Not much, but take that to 100 users ... 1000s?

BioGizzard, you don't need a 'new' format, just compress the new things with 7z, the old things can stay rar'ed till the day your 10 terrabyte flash platter screams for mercy... 7Zip supports decompressing rar and several other formats (same as rar now supports decompressing 7Z). But if you're happy, why change?

Also 7-Zip is as much a new format as Rar is. Both are somewhat similar to Zip, but there's only so many ways to peel a banana (or however it goes).

Used algoritms:
Zip: LZH + LZW + SF + Huff
Rar: f + LZ77 + PPMII + Huff
7Z: f + LZ77 + PPMII + LZMA + BWT

A significat improvement on 7-Zip's performance should probably come out with version 5. Rar uses 'multimedia filtering' to decide if it will be worthwhile to compress a file, something which 7-Zip still lacks. This cuts down a lot on time consuming compression of uncompressable files (eg. mp3's/ogg's/avi/jpg).

Usefull info can be found on http://www.maximumcompression.com/

[Pyro]

A quick bit on sizes - on my wanna-be-56k, it takes about 6 minutes for a meg, so I could've downloaded patch 3355 in 10 minutes less with 7Z ultra than with Rar, and a whole 24 minutes less than the .exe one you get from epic.

77 minutes -vs- 63 min -vs- 53 minutes ...

And it takes less than 5 minutes to decompress and recompress...

[Pyro]

(Yes i know i should learn to shut up)

If you were using compression to compress a blob of data, client side, to get info onto you database you could save quite a bit of precious server performance, by utilizing some of your client's mostly unused computing power... not much, but once again, every bit could help. Obviously only in certain cases yarra yarra ...

[Zyron]

How you doing Pyro ??
How you doing disinnergrater Bio ?

shoosh, im bored ^_^

[Shrimp]

Wow, Pyro's on a real mission :D.

As I said though, personal preference. From my findings so far, RAR will compress on average 48% faster, but give you an average of 4% larger files. This is 7z "Normal" (with default dictionary of 2MB, didn't change any settings here), and WinRAR's "Best" (no settings changed there either).

I guess the perfromance differs between machines as much as the preference differs between people though :p. These findings were obtained on a Pentium M 1.73GHz laptop with 512MB RAM and a 5400 RPM hard disk (included since disk access and write speeds is obviously also a factor).


But *whatever* :p

[<A>BioGizzard]

ola biego zyron
¿como esta?

[sneaky]

and what about PK-Zip?

[Zyron]

Deseo ir detrás los juegos del hogar y del juego

[Peon]

Shot Pyro, explained alot there.

Must agree also with that idea of personal preference.

[Pyro]

Personal preference :)

I just stick to 7zip coz I tend to run out of space more often than processing time ... Hey, Packrats are people too!

Good Good Z, still going strong :)

[VirtualForce]

Been using WinRar for ages. Before that WinZip.
Personally I like the layout and usage of winrar, as you said, its only personal pref, no idea what goes on behind these proggies! :P