Nature of databases

Support related problems and questions

Moderators: TerryRogers, Max

Post Reply
poet-man
Guru
Posts: 630
Joined: Thu May 04, 2017 11:38 am
Has thanked: 10 times
Been thanked: 31 times

Nature of databases

Post by poet-man »

Hi support,

I would like to ask a question in generell:
What I understand EPIM uses Firebird databases. "Databases" is not a subject I'm familiar with.
What I see, however, is, that questions concerning problems with database, are asked rather often. Is this the nature of databases or just Firebird databases, that quite a few cases are described, which have to be solved by creating a new database and transfering data.
To normal users, this seems to suggest, that the used databases do not have the desired steadyness.
The question, which kind of database is the best, would be the same, as the question, which car is the best. Of course, EVERY car is the best.
But I think, the question is allowed to be asked, whether Astonsoft thinks, that there might be a database-solution that brings more steadyness?
At least I could imagine, that it is not the best productivity for user/companies, to have problems with databases and spend manpower to solve them.
What is Astonsoft's point of view about this?

Thanks for giving me some aspects to understand, what this is about.
Everything I mention comes out of my experience. I'm not an expert. Feel free to try anything I suggest ... but always at your own risk.
Frank-H
Experienced
Posts: 57
Joined: Thu Dec 29, 2005 3:28 am
Has thanked: 40 times
Been thanked: 4 times

Re: Nature of databases

Post by Frank-H »

Good questions, poet-man, and yes "the database being trustworthy" is VITAL. If all we needed was a very simple address book, "nothing more important than that", google contacts or any cheap free address book would do. WE want a stable, trustworthy, database.

Regarding that, does anyone know, "if you make a database, contacts / fields / groups", in a stable version of epim, and you later go to an unstable, buggy version that crashes, does it affect that database? (I know, keep lots of backups). But have others found their database "messed up", or is it separate from "the program code" such that the database itself remains intact without damage?
MetalDrop
Guru
Posts: 866
Joined: Sat Apr 09, 2016 10:19 pm
Been thanked: 189 times

Re: Nature of databases

Post by MetalDrop »

Some EPIM updates can certainly damage the database.

However this isn't unique to EPIM nor something they can test reliably for. In most cases and I'd strongly recommend making backups of all important files before opening them in new software updates.

Microsoft and other big companies have fried my access databases, docs, spread sheets, videos...etc countless times with buggy updates.

Bugs that cause damage to files may only effect 1% of people or less making even rigorous testing ineffective at ridding them out before releasing the software to a stable branch, and is the reason I'm extremely against Microsoft's near mandatory updates on widowers 10.

Windows updates have done irreversible damage to quite a lot of files on my computers over the decades, and I ensure I have everything important backed up before I do even a minor update.
EPIM Portable Pro Running/Tested On: Windows 11 Pro 64-bit US-ENG|i5-6400+Quadro P620|i7-7700K+1050ti|i7-8700K+970GTX|AMD 5600x+1080ti|16GB+RAM&NVMe SSDs
[I'm helpful and often reply to questions, however I am just a fellow user and not staff.]
MetalDrop
Guru
Posts: 866
Joined: Sat Apr 09, 2016 10:19 pm
Been thanked: 189 times

Re: Nature of databases

Post by MetalDrop »

poet-man wrote: Mon Jul 12, 2021 11:15 am Hi support,

I would like to ask a question in generell:
What I understand EPIM uses Firebird databases. "Databases" is not a subject I'm familiar with.
What I see, however, is, that questions concerning problems with database, are asked rather often. Is this the nature of databases or just Firebird databases, that quite a few cases are described, which have to be solved by creating a new database and transfering data.
To normal users, this seems to suggest, that the used databases do not have the desired steadyness.
The question, which kind of database is the best, would be the same, as the question, which car is the best. Of course, EVERY car is the best.
But I think, the question is allowed to be asked, whether Astonsoft thinks, that there might be a database-solution that brings more steadyness?
At least I could imagine, that it is not the best productivity for user/companies, to have problems with databases and spend manpower to solve them.
What is Astonsoft's point of view about this?

Thanks for giving me some aspects to understand, what this is about.
Firebird databases are very stable, and have a long history of heavy use in many industries.

The issue with fixing database problems is often you don't know what the issue is without having people send you the database, most people are unwilling to send you a copy of their database filled with personal information. Therefore making a special patch to fix the issue is nearly impossible and the best solution is to simply have the user move their data to an undamaged one.

You also have the fact that adding endless "patches" to fix databases could also cause much worse problems as you get very complex code that could trigger a patch on a database that didn't need it and be the cause of damage.

Something else to think about:


Most "database" issues are caused because something in the database got messed up.

Example:
Maybe an update had a bugged code and added a character in a integer row, so your weird behavior is being cased by the fact EPIM is looking for a number "0-1000" but is getting "10Z".

This is not the database engines, nor the databases fault.

Most database issues fall into this category of issue where the engine nor database is actually to blame and it was the code being executed on the database that is to blame.

A real world example would be: Say you pour apple juice into a cup, but you missed the cup and poured it on the countertop instead. Was it the cups fault that a mess was made?

Or you successfully poured the apple juice into the cup, but it was spoiled. Was it the cups fault that you didn't check the expiration date?

Overall in all the years I've been using databases, I rarely ever see issues that are actually the result database and engine itself [at least on stable long time tested ones like firebird]. The vast majority of the time it is unintentional user/programmer error.
EPIM Portable Pro Running/Tested On: Windows 11 Pro 64-bit US-ENG|i5-6400+Quadro P620|i7-7700K+1050ti|i7-8700K+970GTX|AMD 5600x+1080ti|16GB+RAM&NVMe SSDs
[I'm helpful and often reply to questions, however I am just a fellow user and not staff.]
poet-man
Guru
Posts: 630
Joined: Thu May 04, 2017 11:38 am
Has thanked: 10 times
Been thanked: 31 times

Re: Nature of databases

Post by poet-man »

Very well discussed indeed, I think.
Sounds, that rarely the database is the reason.
Seems a stable DB is like a piece of hardware: Testet lots of times. But as soon as an environment meets it, very many different situations come together, which do something unexpected to the hardware.
So, there is a DB, serving data. And there is hardware, software, user etc.
And it seems, it's all about getting that consistant DB back again, after environment has done harm to data.
So ... user cutting off power, memory unstable, software giving wrong command ....
And the most unstable item is the user ... something I learned, when being member of support unit. But of course, it's easier to get somebody else, who did it, when problem occurs.
And the most important of all is "support", "getting consistant DB back again, as I said. Something I donot say, because I was member of it, years ago, but seems that Astonsoft delivers an excellent piece of it.
So, the empression I had, of possibly using unstable DBs, most likely is unfair, because there are lots of environmental things arround, that disturb performance. And a good support does an excellent job, to keep things running.
Everything I mention comes out of my experience. I'm not an expert. Feel free to try anything I suggest ... but always at your own risk.
Frank-H
Experienced
Posts: 57
Joined: Thu Dec 29, 2005 3:28 am
Has thanked: 40 times
Been thanked: 4 times

Re: Nature of databases

Post by Frank-H »

Very helpful info, MetalDrop and poet-man. In my case, though long ago till now I bought Pro four times but never put it into use (business issues), a week ago buying and creating fields / groups in 9.9.6, there was no "data carried over into the database". There were no odd user errors in this case, it was one person (me) putting new fields in for one contact, "the starting place".

And it all went "crash crash crash" after about 20 custom fields created (simple "names of family" "what time zone" etc, saving after each field, taking it slow...) A self-contained environment so to speak, without many variables. Same (one contact) simple database running fine, with a "from the zendesk support page" 2020 version 9.3, AND, running fine on a program file email support sent me, an "updated" 9.9.6, no crashes.

And support said, "there are known issues" with 9.9.6, "an update will be posted, when the update / fixes are done". I would have to think that better testing would have caught the issues with 9.9.6. This cost me a lot of time, uninstalling / re-installing etc trying to make simple use of newly bought program "work", it was program flaws the whole time. Now getting fixed though.
a8907433
Guru
Posts: 1047
Joined: Fri Mar 12, 2010 11:57 pm
Been thanked: 170 times

Re: Nature of databases

Post by a8907433 »

Hello,
I´m having too errors with 9.9.5.and 9.9.6. I reported this, but it is not critical; I kill EPIM after first start, open again, then it works. I know, in the next version this will be gone, I had this many times.
Advice from Astonsoft would be: Synchronize to a new database.....

Now, with your explanation of databases MetalDrop (Thank you!!), I tried finally to synchronize my many years old database to a new one.

Ok, so far, so good. I have apparently a working database.
BUT: @Admin, Max, Astonsoft.... MetalDrop
1) In NOTES, the notes are arranged by NAME- not by my custom order- it would cost me many hours of sorting, if I want to use this database.
2) in Database Management, the number of entries are identical: Appointments, Tasks, Notes Contacts, Passwords, E-mail messages. But the SIZE of the database shrinked from 587 to 487Mb! I marked "remove double entries".... but which one were the double entries? If I would work with the new database, I can´t revert to the old one (because of daily added entries)- so, with many thousand entries (especially in mail (PoP3), notes, and appointments in calendar), what if in, say, 3 months I find something missing? 120 Mb of data are.... vanished....? How can I trust, that this is the same as my original database? (the original DB was optimized)

Why can this task not be done by Database Management? Optimize database would suggest, that it removes errors in database!

But, as I said before- the biggest obstacle of using the new database - even if you guarantee me, that databases are identical- is the wrong order in the notes module.

To the discussion in this thread, the buggyness of new EPIM versions- caused in most cases by "mistakes" in the database- this could completely be avoided if the database optimize function had synchronizing abillities ("EPIM has recognized errors in db, a new identical db will be save with the same name xyz_OPTIMIZED")

EPIM 9.9.6, synchronizer 9.0, WIN 8.1
Frank-H
Experienced
Posts: 57
Joined: Thu Dec 29, 2005 3:28 am
Has thanked: 40 times
Been thanked: 4 times

Re: Nature of databases

Post by Frank-H »

a8907433, if you have recently system-made backups, (I set mine to make a backup every 6 hours, and keep 10 at a time), you might try opening a past backup, in a program version other than 9.9.6

"Whatever was good and stable for you" before the 9.9.6 issues you described. I made a "known location for epim database files", folder, on my data partition of the PC, and a separate folder named (via tools / options for backups), where auto-backups are put. Finding your main program directory there is an uninstall.exe there, "running that" then installing an earlier version that worked well (I'm keeping program version files as well) might resolve the damage to notes etc you described.
a8907433
Guru
Posts: 1047
Joined: Fri Mar 12, 2010 11:57 pm
Been thanked: 170 times

Re: Nature of databases

Post by a8907433 »

Thank you, Frank-H
I have enough backups, on PC, my USB flashdrive, on a extern server and in EPIM cloud. Also I have most versions from EPIM 3.2 on.
But, it is the WORK-TIME, every time to try out.... So, I will update to next version (9.9.x, 10.0....), and likely most errors are gone (and new ones appearing ;-) ).
But, for THIS kind of "bug"- caused by the DB itself, it would be the most satisfying experience if a single click on "optimize" would eliminating mot errors and the user can work as usual.
MetalDrop
Guru
Posts: 866
Joined: Sat Apr 09, 2016 10:19 pm
Been thanked: 189 times

Re: Nature of databases

Post by MetalDrop »

a8907433 wrote: Tue Jul 13, 2021 8:17 pm Ok, so far, so good. I have apparently a working database.
BUT: @Admin, Max, Astonsoft.... MetalDrop
1) In NOTES, the notes are arranged by NAME- not by my custom order- it would cost me many hours of sorting, if I want to use this database.
2) in Database Management, the number of entries are identical: Appointments, Tasks, Notes Contacts, Passwords, E-mail messages. But the SIZE of the database shrinked from 587 to 487Mb!

Two things spring to mind.

#1 Syncing doesn't sync trash I don't think*. So your deleted items are not in the new database, that could be quite a bit of space if you never emptied your trash before.

#2 Depending on how old the database is, there was an issue back in... 6x~7x... ish days I want to say, where EPIM didn't always properly delete trash and when I had a database that synced from 1GB down to 700MB and I went looking into what was missing using a database editor it was a bunch of old trash that EPIM said was deleted, but it never actually cleared from the database, once I cleaned the corrupted trash out with Razor SQL the two databases matched in size.
EPIM Portable Pro Running/Tested On: Windows 11 Pro 64-bit US-ENG|i5-6400+Quadro P620|i7-7700K+1050ti|i7-8700K+970GTX|AMD 5600x+1080ti|16GB+RAM&NVMe SSDs
[I'm helpful and often reply to questions, however I am just a fellow user and not staff.]
a8907433
Guru
Posts: 1047
Joined: Fri Mar 12, 2010 11:57 pm
Been thanked: 170 times

Re: Nature of databases

Post by a8907433 »

It seems, there is another bug in synchronizer 9.0: I marked "encrypt database" on both sides- but the new db was not encrypted. That explains some of the size of db.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: Nature of databases

Post by admin »

There's no such option to encrypt a database with a password in EPIM Synchronizer. The password field is used to get access to your data if it was password protected. Encryption is always done from EPIM.
Android version of EssentialPIM. Keep all your data in sync!
Post Reply