Search and Repeated Items Speed

Support related problems and questions

Moderators: TerryRogers, Max

GetOrganised
Novice
Posts: 16
Joined: Sun Jul 05, 2020 11:23 am
Has thanked: 3 times

Search and Repeated Items Speed

Post by GetOrganised »

I've been using EssentialPIM for nearly 2 years now and am generally very happy with it, having changed from Palm. However, one aspect that is horrifically worse than Palm on my system is the speed of searches and adding repeated items. If I want to search for a phrase anywhere in the calendar (past and future), it literally takes minutes, as in 5-10 minutes. The Palm used to do it in a second! Similarly, when I create a new repeating Calendar entry it takes the same amount of time to process it, during which the application is locked up.

Am I doing something wrong? Is there a setting I haven't clicked? Does anyone else have this problem? Any help much appreciated.

Finally, this is my first post, so apologies for any protocol violations! I'm guessing this may have come up before, but I had a search around the forums and couldn't find an answer.
Intel i7-4770K 3.5GHz 64-Bit. 8GB memory. SSD disk. Win 7 Pro. Usually on latest EssentialPIM version.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: Search and Repeated Items Speed

Post by admin »

How many events are there in your database file? You can get the number by clicking on File->Database Management. Thanks!
Android version of EssentialPIM. Keep all your data in sync!
GetOrganised
Novice
Posts: 16
Joined: Sun Jul 05, 2020 11:23 am
Has thanked: 3 times

Re: Search and Repeated Items Speed

Post by GetOrganised »

Hi. Thanks for your response. Events as follows:

Appointments: 129172
Tasks: 104
Notes: 5
Contacts: 180

I was quite surprised at the number of appointments! But they do go back to 2002. I like to keep the old ones as a kind of diary record. But even so, 129172 appointments is an average of 20 a day and there certainly aren't that many. So I'm not sure what that number is counting. Either way, there were the same number in my old Palm and it searched real quick.
Intel i7-4770K 3.5GHz 64-Bit. 8GB memory. SSD disk. Win 7 Pro. Usually on latest EssentialPIM version.
GetOrganised
Novice
Posts: 16
Joined: Sun Jul 05, 2020 11:23 am
Has thanked: 3 times

Re: Search and Repeated Items Speed

Post by GetOrganised »

Hi. I just did a quick test. Using Advanced Search looking for a single word, excluding deleted, Calendar only, Subject field only in the last 12 months only. This took 3.5 minutes to return results.

Hope that helps.
Intel i7-4770K 3.5GHz 64-Bit. 8GB memory. SSD disk. Win 7 Pro. Usually on latest EssentialPIM version.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: Search and Repeated Items Speed

Post by admin »

Thanks, normally it should be happening much faster with that number of events. What's the first EPIM version the database was opened in? You can check that again in File->Database Management. If the version number is less than 8.x, I'd recommend you to sync your database to the new one using EPIM Sync (https://www.essentialpim.com/pc-version ... ntary-apps).
Android version of EssentialPIM. Keep all your data in sync!
GetOrganised
Novice
Posts: 16
Joined: Sun Jul 05, 2020 11:23 am
Has thanked: 3 times

Re: Search and Repeated Items Speed

Post by GetOrganised »

Hi. The first version number is 8.03. I just ran another test - new appointment, just a single word of text, all day event, with reminder, repeat every week. This took exactly 3 mins to process, with the application locked up while it was doing it.
Intel i7-4770K 3.5GHz 64-Bit. 8GB memory. SSD disk. Win 7 Pro. Usually on latest EssentialPIM version.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: Search and Repeated Items Speed

Post by admin »

Please describe in more details and preferably step by stat what you specifically you do. Is that a quick search at the top or an advanced search? if the latter, what's the search settings?
Android version of EssentialPIM. Keep all your data in sync!
GetOrganised
Novice
Posts: 16
Joined: Sun Jul 05, 2020 11:23 am
Has thanked: 3 times

Re: Search and Repeated Items Speed

Post by GetOrganised »

Searching example:

Tools > Advanced Search
Look in: Calendar
Include deleted items un-checked.
General tab:
Enter search word in 'Search for' box.
Search in fields: Subject
Time checkbox ticked.
Created From: 13/07/2019 To: 13/07/2020
No changes to Advanced or Replace tabs.
Click 'Find now'.

This took exactly 3 minutes to the second and returned 4 results.

Repeated item example:

Start and end dates set to today.
Duration: 1 day.
'All day event' checked.
'Exclusive appointment' unchecked.
No Location.
Set to my 'Personal' category.
No Priority.
Completion: None.
Reminder: 0 minutes.
'Repeat every' checked. Set to 1 week.
'Expires' unchecked.
Uses my 'Personal' template.
A single word in 'Event Name'.
Set as 'Public'.
'Read Only' unchecked.
No Related Items.
No Tags.
Click 'OK'.

Bizarrely, this also took exactly 3 minutes to the second??!!

Hope this helps.
Intel i7-4770K 3.5GHz 64-Bit. 8GB memory. SSD disk. Win 7 Pro. Usually on latest EssentialPIM version.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: Search and Repeated Items Speed

Post by admin »

Thanks, will check what can be done in order to speed search up (although, the main point here would be to reduce the number of events using EPIM Archiver). And will also fix the issue that takes too much time when creating new events.
Android version of EssentialPIM. Keep all your data in sync!
MetalDrop
Guru
Posts: 866
Joined: Sat Apr 09, 2016 10:19 pm
Been thanked: 189 times

Re: Search and Repeated Items Speed

Post by MetalDrop »

I'm curious what is your CPU and RAM load before and during you searches?
Also what is the specs of the hard drive you have your database on and have you checked it's SMART info lately to make sure it's healthy?

My biggest database is quite a bit more data heavy then yours is and I can't get an advanced search to take longer than a few seconds.
EPIM Portable Pro Running/Tested On: Windows 11 Pro 64-bit US-ENG|i5-6400+Quadro P620|i7-7700K+1050ti|i7-8700K+970GTX|AMD 5600x+1080ti|16GB+RAM&NVMe SSDs
[I'm helpful and often reply to questions, however I am just a fellow user and not staff.]
GetOrganised
Novice
Posts: 16
Joined: Sun Jul 05, 2020 11:23 am
Has thanked: 3 times

Re: Search and Repeated Items Speed

Post by GetOrganised »

Hi and thanks to all for their attention so far. My database is on a SSD disk and according to my SSD monitoring utility it's healthy. I used Windows Performance Monitor to see what happens when I kick off a search. The CPU shoots up to 100% and stays there, EPIM goes to the top of the list and after a bit it's listed as not responding, not much network or disk activity, memory usage for EPIM progessively increases up to about 321MB. When the search completes, everything reverts to normal. I installed EPIM and my DB on a laptop this morning and a search as above took 4 mins 20 secs, so I don't think it's my PC.

I tried a few other things. Running Optimize on Database Management improved the search time by about 30 secs.

If I switch to year view whilst in this year, it takes about 14 secs to populate. If I do it in a previous year, it populates almost immediately. Ie if I'm in Day view set to 1st Jan 2020 it takes 14 secs to switch to year view and populate it. If I go back 1 day to 31st Dec 2019, it's immediate. I don't think it's a problem with an entry I've put in this year, though, as it was just as slow last year!

If I switch to table view and set the From date to 2002 (my earliest calendar entries), it takes about 5 secs to display all the Calendar entries going back to 2002, 40512 entries displayed. If I type a search term in the Search in: box at the top right, results are immediate. This is actually the fastest way I've discovered to search the entire Calendar.

Finally, I imported my entries from a PalmOne back in 2018 when I started using EPIM, if that helps.
Intel i7-4770K 3.5GHz 64-Bit. 8GB memory. SSD disk. Win 7 Pro. Usually on latest EssentialPIM version.
Keldi
Guru
Posts: 377
Joined: Thu Aug 23, 2012 11:42 am
Has thanked: 26 times
Been thanked: 92 times

Re: Search and Repeated Items Speed

Post by Keldi »

GetOrganised wrote: Tue Jul 14, 2020 2:56 pm If I type a search term in the Search in: box at the top right, results are immediate.
I see the difference between the Advanced Search and Quick Search even on a relatively small database (10457 calendar entries).

If I search for something rare (resulting in only 1 item found in my database), the response is fast in both, but the Advanced Search takes a couple of seconds more.
But if I try to search for some common word, the Advanced Search takes some time.

For example, I searched for the word and in all my Calendar entries.

Calendar->Table view ('from date' unchecked to show all entries and no filters applied, Quick Search -> All Fields; in the table all columns are shown except for the Notes).
Time: Immediate or close to it (at least I didn't have time to start a stopwatch after I typed 'and' as the results appeared fast)
3979 items found

Advanced Search (General Tab, Look in: Calendar, Search in fields: Subject, body and tags; columns in the results table: Item type, Subject, Created, Item location, Category, Tags)
Time: 20 sec
3979 items found

However,
Advanced Search, the same parameters, but the results table is customized to show only one column that isn't even for appointments (the right mouse click on the column caption, Customize, << button, All Mail fields, Starred, > button, Ok)
Time: 6 sec! (and then right-mouse click, customize, reset - and I have the exact same table with results as in my initial advanced search but in less time even with all the extra customizing actions).


@GetOrganised
Please try to do this one
GetOrganised wrote: Mon Jul 13, 2020 10:27 am Searching example:

Tools > Advanced Search
Look in: Calendar
Include deleted items un-checked.
General tab:
Enter search word in 'Search for' box.
Search in fields: Subject
Time checkbox ticked.
Created From: 13/07/2019 To: 13/07/2020
No changes to Advanced or Replace tabs.
Click 'Find now'.

This took exactly 3 minutes to the second and returned 4 results.
with the same searched word you used before but with customizing the columns in the results table to show only one column before hitting the Find Now button. I wonder whether it makes a difference bigger than my 14 sec on a bigger database.
But I might have found a totally separate issue with speed.

@admin
I think it's sill worth looking into even if it'll end up only slowing down the search for not much more than 10 seconds in some occasions. The table view in Calendar has no problems with showing results immediately even though there are more columns there. I expect the Advanced Search to take a bit longer as it specifies the fields to search in. But It's really bizarre that I can get the results faster by manually unchecking columns and switching them back on after the search is done. :o
MetalDrop
Guru
Posts: 866
Joined: Sat Apr 09, 2016 10:19 pm
Been thanked: 189 times

Re: Search and Repeated Items Speed

Post by MetalDrop »

Some interesting things I came across:

#1 My screen DPI is 150% on windows 10 and the result numbers in advanced search are being cut off.

#2 The quick search and advanced search are given concerning different results.
- Quick search returned 13285
- Advanced with the same settings returned only 13076

I did notice that speeds where slightly different but not that noticeable on my main system.
- The quick search took less than a second.
- The advanced search took slightly longer than one second.

However when I took the test to an older system Intel 2600k system:
- The quick search took less than a second.
- The advanced search took about 5 minutes.

- So it would seem that the algorithm used in advanced search is getting a good boost from my hardware.
Most likely either:
Intel 2600k vs Ryzen 2600
or
Samsung SATA SSD vs Samsung NVMe SSD
Attachments
2020-07-14_233701.png
quick.png
EPIM Portable Pro Running/Tested On: Windows 11 Pro 64-bit US-ENG|i5-6400+Quadro P620|i7-7700K+1050ti|i7-8700K+970GTX|AMD 5600x+1080ti|16GB+RAM&NVMe SSDs
[I'm helpful and often reply to questions, however I am just a fellow user and not staff.]
GetOrganised
Novice
Posts: 16
Joined: Sun Jul 05, 2020 11:23 am
Has thanked: 3 times

Re: Search and Repeated Items Speed

Post by GetOrganised »

Thanks for all the comments.

Keldi - How do I customize the results table in Advanced Search? On mine, the results table is only configurable after the search has completed.

MetalDrop - I tried a search for a term in table view going back to my earliest entries (2002) and got 3860 results. I then tried searching for the same term with Advanced Search and got 62 results! The difference seems to be that Advanced Search returns only the first entry of a repeating series. Table returns them all, which is much easier to see when the next repeating entry is due.

I have rapidly come to the conclusion that Table view is the place to go for searches. But that doesn't resolve my issue of the time it takes to enter a new repeating entry.
Intel i7-4770K 3.5GHz 64-Bit. 8GB memory. SSD disk. Win 7 Pro. Usually on latest EssentialPIM version.
Keldi
Guru
Posts: 377
Joined: Thu Aug 23, 2012 11:42 am
Has thanked: 26 times
Been thanked: 92 times

Re: Search and Repeated Items Speed

Post by Keldi »

GetOrganised wrote: Wed Jul 15, 2020 11:39 am How do I customize the results table in Advanced Search? On mine, the results table is only configurable after the search has completed.
Sorry, I wasn't paying attention that it doesn't show the table before the first search. I started to experiment with columns after I already have searched with the default settings.
So, it'll be:
  • Start a search with Find Now button
  • Stop button
  • Customizing columns
  • Trying a new search with less columns
It's worth a try, but I don't hold much hope that it'll ever outspeed the Calendar Table View search. The latter is really the best place for a search for the time being.
Post Reply