FreeArc Compression + .EPIM = Magic

General talks about EssentialPIM

Moderators: TerryRogers, Max

Post Reply
FireBrand2026
Guru
Posts: 142
Joined: Fri May 13, 2016 4:10 am
Been thanked: 9 times

FreeArc Compression + .EPIM = Magic

Post by FireBrand2026 »

Today I decided to try compressing my EPIM backups into the .Arc file format using PeaZip.

20GB of backups compressed to 100MB.

Didn't believe it, so I copied the archive to another PC extracted it and tested random samples in the backup list and they all worked flawlessly.

I just wanted to let others know about this, and possibility the EPIM staff as well, because this is truly amazing. I had in the past just used 7zip ULTRA and it knocked the 20GB down to 10GB and took forever doing so.

A friend of mine was telling me how amazing the FreeARC compression format was for text and I wanted to try it out myself so that's why I did what I did since it was the biggest ball of text I could think of on my computer.
FireBrand2026
Guru
Posts: 142
Joined: Fri May 13, 2016 4:10 am
Been thanked: 9 times

Re: FreeArc Compression + .EPIM = Magic

Post by FireBrand2026 »

Update:

Tried this with my Email database backups and it didn't work as magically, just about the same as 7zip, but I was warned that it only worked well on text.

Still amazed at how well it handles my note databases. I mean yes in theory there is only 100MB of unique data spanning hundreds of backups so I shouldn't be super surprised, but I had tried every compression type 7zip offered at the highest settings my system could handle and nothing ever came close to this.
a8907433
Guru
Posts: 1047
Joined: Fri Mar 12, 2010 11:57 pm
Been thanked: 170 times

Re: FreeArc Compression + .EPIM = Magic

Post by a8907433 »

Did you compare it with the comperession built IN EPIM (File/Database Management/Optimize)?
FireBrand2026
Guru
Posts: 142
Joined: Fri May 13, 2016 4:10 am
Been thanked: 9 times

Re: FreeArc Compression + .EPIM = Magic

Post by FireBrand2026 »

a8907433 wrote: Mon Jun 01, 2020 9:33 am Did you compare it with the comperession built IN EPIM (File/Database Management/Optimize)?
I did this for bulk backups not standard single file backups.


That said I went ahead and tested putting a single uncompressed epim file into an ARC and than compared it to the compressed backups and it is 30% smaller.

Using Arc on a single pre-compressed EPIM backup also cut the size in half resulting in an even smaller file.
Max
Site Admin
Posts: 21714
Joined: Wed Dec 08, 2004 11:39 pm
Has thanked: 819 times
Been thanked: 364 times
Contact:

Re: FreeArc Compression + .EPIM = Magic

Post by Max »

Different compression algorithms will compress data differently, but generally text data is well-packable. Usually compression comes at the cost of speed, hence most often used data is not compressed (or compressed lightly)
Maxim,
EPIM Team
FireBrand2026
Guru
Posts: 142
Joined: Fri May 13, 2016 4:10 am
Been thanked: 9 times

Re: FreeArc Compression + .EPIM = Magic

Post by FireBrand2026 »

Max wrote: Mon Jun 01, 2020 7:57 pm Different compression algorithms will compress data differently, but generally text data is well-packable. Usually compression comes at the cost of speed, hence most often used data is not compressed (or compressed lightly)
Which is one of the amazing things about ARC is that it seems to be super FAST at least on my system. 20GB to 100MB in under 5mins which was even faster than just zipping the files at level 1 compression.

When compressing single databases at the highest setting the window barely even blinked on screen.

Which of course I wouldn't expect this to ever be used for the active database I think it should stay uncompressed, but it could be integrated into the backup system and save people massive amounts of data well letting them keep huge numbers of backups in the same space they could have only kept a couple.

Mainly I just wanted to let other uses know about it since ARC is a nearly unknown compression type and as I said I tried everything 7zip offers which is the most common, at the highest most painful settings possible and nothing even came into the same ballpark.
MetalDrop
Guru
Posts: 866
Joined: Sat Apr 09, 2016 10:19 pm
Been thanked: 189 times

Re: FreeArc Compression + .EPIM = Magic

Post by MetalDrop »

Great find!

Took 57 seconds for me to compress 10GB of note backups into 80MB
EPIM Portable Pro Running/Tested On: Windows 11 Pro 64-bit US-ENG|i5-6400+Quadro P620|i7-7700K+1050ti|i7-8700K+970GTX|AMD 5600x+1080ti|16GB+RAM&NVMe SSDs
[I'm helpful and often reply to questions, however I am just a fellow user and not staff.]
S_Belle
Guru
Posts: 195
Joined: Sun Sep 15, 2019 4:54 pm
Been thanked: 4 times

Re: FreeArc Compression + .EPIM = Magic

Post by S_Belle »

50GB of backups to 500MB here. I didn't pay attention to how long it took, but it was less time than it took me to eat a quick dinner.
User avatar
mvisconte
Beta Team
Posts: 236
Joined: Fri May 06, 2005 1:10 am
Location: East-west quadrant, Thyroid Nebula
Has thanked: 50 times
Been thanked: 15 times

Re: FreeArc Compression + .EPIM = Magic

Post by mvisconte »

a8907433 wrote: Mon Jun 01, 2020 9:33 am Did you compare it with the comperession built IN EPIM (File/Database Management/Optimize)?
I wasn't aware that Optimize featured compression. I think that it just removes deleted items from the database, resulting in a smaller, "tighter" database.
MetalDrop
Guru
Posts: 866
Joined: Sat Apr 09, 2016 10:19 pm
Been thanked: 189 times

Re: FreeArc Compression + .EPIM = Magic

Post by MetalDrop »

mvisconte wrote: Fri Jun 12, 2020 2:20 am
a8907433 wrote: Mon Jun 01, 2020 9:33 am Did you compare it with the comperession built IN EPIM (File/Database Management/Optimize)?
I wasn't aware that Optimize featured compression. I think that it just removes deleted items from the database, resulting in a smaller, "tighter" database.
Optimizing doesn't compress to my knowledge, but the backup databases generated by the auto-backup system in EPIM are compressed.
EPIM Portable Pro Running/Tested On: Windows 11 Pro 64-bit US-ENG|i5-6400+Quadro P620|i7-7700K+1050ti|i7-8700K+970GTX|AMD 5600x+1080ti|16GB+RAM&NVMe SSDs
[I'm helpful and often reply to questions, however I am just a fellow user and not staff.]
User avatar
mvisconte
Beta Team
Posts: 236
Joined: Fri May 06, 2005 1:10 am
Location: East-west quadrant, Thyroid Nebula
Has thanked: 50 times
Been thanked: 15 times

Re: FreeArc Compression + .EPIM = Magic

Post by mvisconte »

MetalDrop wrote: Fri Jun 12, 2020 2:36 am
Optimizing doesn't compress to my knowledge, but the backup databases generated by the auto-backup system in EPIM are compressed.
That verifies my experience. I hadn't measured it before, but I do see about a 20% reduction in the size of the backup files.
FireBrand2026
Guru
Posts: 142
Joined: Fri May 13, 2016 4:10 am
Been thanked: 9 times

Re: FreeArc Compression + .EPIM = Magic

Post by FireBrand2026 »

I was messing around today and found ZPAQ

http://mattmahoney.net/dc/zpaq.html

It's even more magical as it works on encrypted Email heavy Databases as well!

I figured I bring it up here as a nudge for the EPIM devs to look into it.

Zpaq is damage resistant, handles duplicate exceedingly well, and allows very easy rollback and versioning of backups via command line.

If it was built into the EPIMs database auto backup system it could allow relatively limitless backups of databases, and with a tiny UI built for it in EPIM, would provide a nice User experience for database recovery and rollbacks.

Screenshot: 3x encrypted email database backups taking up over 3GB compressed to Zpaq in less than 1 Minute, and takes of less space than a single backup.
2020-11-20_145713.png
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: FreeArc Compression + .EPIM = Magic

Post by admin »

Thanks for the info, will have a look at it.
Android version of EssentialPIM. Keep all your data in sync!
Post Reply