HUGE file size

Support related problems and questions

Moderators: TerryRogers, Max

Post Reply
kenknight
Novice
Posts: 20
Joined: Thu Feb 18, 2010 3:30 pm

HUGE file size

Post by kenknight »

Hi All,

I've been using EPIM for several years. Primarily for password and note storage. I just noticed that the epim file went from about 4megs to 2.8gigs between 2018 and 2019. What the heck? I think something has run amuck here.

Also I just noticed that the hierarchy of my passwords has been all screwed up. It doesn't appear I lost any, however, if I look at older versions there are folder groups whereas many of them are missing in the later version.

Is this something I'm going to have to get support to handle or is there a repair process?

thanks,
Ken
MetalDrop
Guru
Posts: 866
Joined: Sat Apr 09, 2016 10:19 pm
Been thanked: 189 times

Re: HUGE file size

Post by MetalDrop »

Large file size can be perfectly normal depending on what you are doing. A well used pop3 email account for example were you rarely delete anything or never empty your trash can quite easily build up 1-5gigs in a years time.

Some things that can help if you think you might actually be having troubles:
On the download page: https://www.essentialpim.com/get-epim
There is Archiver/DupeRemover & EPIM Synchronizer

Archiver/DupeRemover is helpful for getting rid of duplicates that were caused by Cloud syncing errors or other weird glitches. It can also be helpful in archiving old data you don't need any longer, but are hesitant to fully delete.


EPIM Synchronizer is extremely helpful in that it lets you move you're entire database contents to a new database file which can repair odd update damage that can occur and result in all sorts of odd bugs.

I've found for myself I normally need to Sync to a new database file at least every 2 major versions of EPIM.

At this time if you go to File > Database Management and at the button you see that the database has been opened with a version less than 8.5x I'd personally recommend syncing the database to a new database file.

In the same dialog box before you Sync to a new file I'd recommend running the Database Optimizer and clearing any unused synchronization data.

Note: Synchronizer will only put the current contents of your database as they are into a new database file any "sorting or organization" damage will have to be fixed by hand.
EPIM Portable Pro Running/Tested On: Windows 11 Pro 64-bit US-ENG|i5-6400+Quadro P620|i7-7700K+1050ti|i7-8700K+970GTX|AMD 5600x+1080ti|16GB+RAM&NVMe SSDs
[I'm helpful and often reply to questions, however I am just a fellow user and not staff.]
kenknight
Novice
Posts: 20
Joined: Thu Feb 18, 2010 3:30 pm

Re: HUGE file size

Post by kenknight »

so I think I figured it out. Somehow the thing was syncing some google drive stuff. What the heck. I don't remember ever telling it to sync google drive. I'll have to investigate further why it did that, but at least I've got it down to what is a normal size for me ... 5 megs.

Are you familiar with what would cause the google drive stuff to get added as an attachment without explicitly doing so?

Thanks,
Ken
MetalDrop
Guru
Posts: 866
Joined: Sat Apr 09, 2016 10:19 pm
Been thanked: 189 times

Re: HUGE file size

Post by MetalDrop »

If I were to hazard a guess, perhaps you at some point setup a google sync for your calendar or contacts and didn't realize that by default the checkboxes for Notes & passwords [both use google drive] where checked.
2020-07-14_122132.png
EPIM Portable Pro Running/Tested On: Windows 11 Pro 64-bit US-ENG|i5-6400+Quadro P620|i7-7700K+1050ti|i7-8700K+970GTX|AMD 5600x+1080ti|16GB+RAM&NVMe SSDs
[I'm helpful and often reply to questions, however I am just a fellow user and not staff.]
User avatar
mvisconte
Beta Team
Posts: 236
Joined: Fri May 06, 2005 1:10 am
Location: East-west quadrant, Thyroid Nebula
Has thanked: 50 times
Been thanked: 15 times

Re: HUGE file size

Post by mvisconte »

MetalDrop wrote: Tue Jul 14, 2020 6:57 pm ...
EPIM Synchronizer is extremely helpful in that it lets you move you're entire database contents to a new database file which can repair odd update damage that can occur and result in all sorts of odd bugs.

I've found for myself I normally need to Sync to a new database file at least every 2 major versions of EPIM.

At this time if you go to File > Database Management and at the button you see that the database has been opened with a version less than 8.5x I'd personally recommend syncing the database to a new database file.
Thank you for this.
I had been wondering about the size and slowness of my main database...
It has gotten to the point where moving from one note to another could take 20 seconds to a couple of minutes.

Doing the sync to a non-existing DB has created a new DB, about 60% of the size of the original. I'll be playing with it for a while, but I think that it's already sped up tremendously - based on limited testing... moving a dozen or so notes with subnotes to a new branch. This had been a several minute task, and it's done in less than 30 seconds.

I had only used this to sync between DBs that already spanned multi-versions - this is the first time I've gone straight to NEW.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: HUGE file size

Post by admin »

mvisconte, going back to your old database (the one you synced over to the new one), please open it and click on File->Database Management, copy and paste here the "This database was opened in" string. Would be helpful to see history of the database. Thanks!
Android version of EssentialPIM. Keep all your data in sync!
User avatar
mvisconte
Beta Team
Posts: 236
Joined: Fri May 06, 2005 1:10 am
Location: East-west quadrant, Thyroid Nebula
Has thanked: 50 times
Been thanked: 15 times

Re: HUGE file size

Post by mvisconte »

admin wrote: Fri Jul 17, 2020 4:46 pm mvisconte, going back to your old database (the one you synced over to the new one), please open it and click on File->Database Management, copy and paste here the "This database was opened in" string. Would be helpful to see history of the database. Thanks!
I had wondered about how to do an upgrade... The Sync to (non-existent) EPIM DB did a wonderful treat.
Granted, my EPIM file size was a small fraction of the person w/ the GB+ size db.

The EPIM db before the upgrade:
7.64, 7.66, 8.01, 8.03, 8.05, 8.11, 8.14, 8.5, 8.51, 8.52, 8.53.1, 8.54.1, 8.54.3, 8.55, 8.56, 8.6.1, 8.61.1, 8.62.1, 8.63, 8.65, 8.66.1, 9.0

I have some much older databases (archived now) that went back to... 1.52, if I recall correctly.
lbw2112
Expert
Posts: 78
Joined: Thu Oct 23, 2014 6:47 pm
Has thanked: 16 times
Been thanked: 10 times

Re: HUGE file size

Post by lbw2112 »

MetalDrop wrote: Tue Jul 14, 2020 6:57 pm EPIM Synchronizer is extremely helpful in that it lets you move you're entire database contents to a new database file which can repair odd update damage that can occur and result in all sorts of odd bugs.

I've found for myself I normally need to Sync to a new database file at least every 2 major versions of EPIM.

Thanks for this tip. I wasn't aware of doing that. Mine went back to version 6 since the last time I created a new database file.

Larry
lbw2112
Expert
Posts: 78
Joined: Thu Oct 23, 2014 6:47 pm
Has thanked: 16 times
Been thanked: 10 times

Re: HUGE file size

Post by lbw2112 »

I had a big error. Not sure if it's something I did wrong or have to change in the new file's options or not but I can't figure it out.

I created a new blank database file, exited EPIM and used EPIM Sync to sync everything from my original file to the new file. All seemed fine in the new file but when I was checking it I noticed a lot of my notes not showing. They're showing the tree and Note names fine, but when I select them the note is blank. I also noticed the the Leaf Bar isn't there on the ones that are blank. Some of the notes are there, especially the ones with more than one leaf, and when I select those the Leaf bar shows up again. But there were hundreds that were blank and didn't show the Leaf Bar. It didn't matter what tree I was in, some notes showed but most didn't. I went back to the original file for now.

I thought maybe it was the theme so changed it to Light mode but that didn't show anything different. I had all the modules selected in the Sync options. Is there a setting per note that makes the Leaf bar show or not?

Thanks,
Larry

Edited: Think I may have found something. It's doing this on notes that are Read-Only on the original file.
So do all the notes that are Read-Only have to be unchecked on the original file before doing the Sync? When I went to the new file and unchecked "Read-Only" on some of those notes, the Leaf bar showed up again, even when checking it on again, but the note still wasn't there.
MetalDrop
Guru
Posts: 866
Joined: Sat Apr 09, 2016 10:19 pm
Been thanked: 189 times

Re: HUGE file size

Post by MetalDrop »

lbw2112 wrote: Sun Jul 19, 2020 3:27 am I had a big error. Not sure if it's something I did wrong or have to change in the new file's options or not but I can't figure it out.

I created a new blank database file, exited EPIM and used EPIM Sync to sync everything from my original file to the new file. All seemed fine in the new file but when I was checking it I noticed a lot of my notes not showing. They're showing the tree and Note names fine, but when I select them the note is blank. I also noticed the the Leaf Bar isn't there on the ones that are blank. Some of the notes are there, especially the ones with more than one leaf, and when I select those the Leaf bar shows up again. But there were hundreds that were blank and didn't show the Leaf Bar. It didn't matter what tree I was in, some notes showed but most didn't. I went back to the original file for now.

I thought maybe it was the theme so changed it to Light mode but that didn't show anything different. I had all the modules selected in the Sync options. Is there a setting per note that makes the Leaf bar show or not?

Thanks,
Larry

Edited: Think I may have found something. It's doing this on notes that are Read-Only on the original file.
So do all the notes that are Read-Only have to be unchecked on the original file before doing the Sync? When I went to the new file and unchecked "Read-Only" on some of those notes, the Leaf bar showed up again, even when checking it on again, but the note still wasn't there.
Sounds like it could be a bug. There are options in both the main EPIM security options and EPIM Syncronizer options to allow or disallow private items but not read-only items.

I don't use Syncronizer for notes because it doesn't grab custom icons, and it's really easy to get all your notes from one database to another without it.

I just Ctr+a to select the whole note tree -> copy-> Then Past to the new database [I use two portable copies of EPIM to have both databases open at the same time easily.]

Which the copy->past method above I know works for read-only items.
EPIM Portable Pro Running/Tested On: Windows 11 Pro 64-bit US-ENG|i5-6400+Quadro P620|i7-7700K+1050ti|i7-8700K+970GTX|AMD 5600x+1080ti|16GB+RAM&NVMe SSDs
[I'm helpful and often reply to questions, however I am just a fellow user and not staff.]
a8907433
Guru
Posts: 1047
Joined: Fri Mar 12, 2010 11:57 pm
Been thanked: 170 times

Re: HUGE file size

Post by a8907433 »

EPIM Synchronizer is extremely helpful in that it lets you move you're entire database contents to a new database file which can repair odd update damage that can occur and result in all sorts of odd bugs.
My database is originally from the version 3.23 (portable, pro), and I opened it with (with some db-conversions for some new versions of EPIM) in many (30, 40??) new version of EPIM..... I never used EPIM Synchronizer, only "file/Database Management/Optimize", and it seems, this worked well. So, what is the difference in synchronizing to a new database and optimizing the db?

As lbw2112 wrote, I am afraid, that I lose some data without noting it- It would be very much work to check each note, sub-note, sub-sub-note and panels in notes, folders in emails and so on....- and as lbw2112 wrote, he was missing some data!

I think, database integrity is the very most important quality of a database- there should be only one, 100% reliable method of checking and repairing a database, with no possibillity of losing data by chance!
lbw2112
Expert
Posts: 78
Joined: Thu Oct 23, 2014 6:47 pm
Has thanked: 16 times
Been thanked: 10 times

Re: HUGE file size

Post by lbw2112 »

MetalDrop,

Thanks for the help.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: HUGE file size

Post by admin »

lbw2112, this looks like something we need to fix.

MetalDrop, will also include synchronization of the custom icons from notes in the new version of EPIM Sync, actually this already works with the latest EPIM sync version.

a8907433, when you do optimization, basically you core database structure stays intact, when you perform sync to a new database, the DB structure will be re-created from the scratch. Sync into a new DB might be helpful in some rare cases but usually isn't really needed.The fact that you still run the DB created in 3.23 proves that.
lbw2112 used synchronization and some data wasn't synced due to its read-only status. This has nothing to do with data integrity in the database itself.
Android version of EssentialPIM. Keep all your data in sync!
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: HUGE file size

Post by admin »

BTW there's the updated EPIM Sync version available that fixes the issue with synchronization of read-only leafs in notes. Please download it from here: https://www.essentialpim.com/get-epim
Android version of EssentialPIM. Keep all your data in sync!
Post Reply