Parts of Eramba (for instance, the risk management page) are slow. I mean ridiculously, unimaginably slow.
11.23s to transfer 1.5KB?!!?!?
Is anyone else experiencing these issues? Hopefully, I’ve misconfigured something on my end, and the fix will be quick.
we exchanged a few emails with two customers (out of 200) that had slow response when loading pages in eramba (not saving) - we are pretty sure they have different issues (one has a huge database) but we are still keeping an eye on this and will be doing a bit of testing soon too.
in our experience two things make eramba go slow:
our troubleshooting is basically login at the linux box and running these commands:
root@demo-e:~# cat /proc/cpuinfo (we want to see you have multiple cores, so more than one “processor”)
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
stepping : 2
microcode : 0x3c
cpu MHz : 2394.518
cache size : 30720 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips : 4789.03
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
root@demo-e:~# free -m (you should have 16gb of “physical” ram and swap must be enabled (in the screenshot shown below it does not have swap and that is a big problem)
total used free shared buff/cache available
Mem: 1998 837 421 85 740 1019
Swap: 0 0 0
root@demo-e:~#
root@demo-e:~# df -h (you should have spare space in your disks)
Filesystem Size Used Avail Use% Mounted on
udev 991M 0 991M 0% /dev
tmpfs 200M 21M 180M 11% /run
/dev/xvda1 7.8G 3.6G 3.8G 49% /
tmpfs 1000M 0 1000M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1000M 0 1000M 0% /sys/fs/cgroup
/dev/xvdf 99G 6.7G 87G 8% /var/www
tmpfs 200M 0 200M 0% /run/user/0
root@demo-e:~#
root@demo-e:~# sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync (your disks must be FAST!!! this is what makes the largest difference)
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.9753 s, 82.8 MB/s
root@demo-e:~#
root@demo-e:~# dd if=tempfile of=/dev/null bs=1M count=1024 (reading and writing they must be fast)
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.891 s, 67.6 MB/s
root@demo-e:~#
root@demo-e:~# mysql -u root -p (log into your database)
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 164534
Server version: 5.7.25-0ubuntu0.16.04.2 (Ubuntu)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
mysql> SHOW PROCESSLIST; (run this command, it should return empty or if not the column “time” should not have any item with more than 1 or 2 minutes … if there is something with more then that is a “hang” process killing the db. restart the engine and it will go away)
±-------±-----±----------±-----±--------±-----±---------±-----------------+
| Id | User | Host | db | Command | Time | State | Info |
±-------±-----±----------±-----±--------±-----±---------±-----------------+
| 164534 | root | localhost | NULL | Query | 0 | starting | SHOW PROCESSLIST |
±-------±-----±----------±-----±--------±-----±---------±-----------------+
1 row in set (0.00 sec)
root@demo-e:~# uptime (depending on the number of cores this number can go high, but to keep things simple they should be lower than 2 at all times)
06:38:11 up 268 days, 22:33, 1 user, load average: 0.00, 0.00, 0.00
root@demo-e:~#
The template 2.x changes index by allowing you to load one or more (up to there) filters , this is far heavier than 1.x which had a far simpler approach. If you have many items in a section, load just one filter and make all other filters NOT to load on the default index … this should help.
edit the filters you dont need on the default index of a section:
go to “manage” tab and toggle this option, then save.
On top of that you can remove the columns you dont need from your filters, this will help too, some columns make a huge different in terms of load. You do that by editing the filter and un-ticking the columns you dont need.
Always remember to “save” your changes on the “manage” tab … 
A 2.x index takes anywhere around 2-8 seconds to load , some can be slower (like compliance analysis) but you should be able to control this by playing with the filters and columns you display.
We have slow response in our Third Party pages. We have well in to the thousands of entries. When ever we would try to add a new entry , it would actually take up to around 16 seconds to return the page back to us. We are using a high end virtual Linux server running Ubuntu with lots of memory, cores and very high speed SSD storage. We also have the database and web server on the same machine. We just kept trying to throw more resources at it. It didn’t really seem to change the outcome. What did change the outcome was using filters. Haven’t had time to look in the code to see how the lookups are handled, to prove this. However, if I filtered the third parties and added from a filtered paged, the response became much more manageable. My conjecture is that it something to do with the presentation layer of the framework if a lot of data is returned.
I just retested this while writing this post. I set a filter for a partial name in third parties that I knew would only return 4 to 5 entries. I then added a test third party from the filtered results page. This added the new entry and returned the page to me in around 2 seconds. I then cleared the filter so all entries were returned. I added another new test entry. This caused the system to add the entry but only returned the page to me after 16 seconds. This was repeatable.
This means you try to “Add” something? Slowness adding items is in %99 of the cases a network issue not really a database thing is inserts are light in essence. If adding is slow is because the engine is blocked doing other things (like loading indexes or having “hung” processes).
Right - and filters is were performance can be an issue as we see now in particular if you are in the range of thousands of objects, we know why and how to change things a little bit to improve them … we did not believe it would be necessary to put in place pagination (which is what 1.x filters had) but it seems it is. We will be working next week on changes towards improving things related to filters.
It would help us see the steps over a call - we’ll reach out over support to try and schedule a call to use this test case for further testing if you dont mind.
we meet a customer today, the performance issues is aggravated when using many custom fields (in their case 60 custom fields).
we are about to finish release 2.0.11 (next monday in theory) - the next release we’ll focus on performance hopefully we’ll be able to help.
Those of you with slow performance can you try this: Performance and FastCGI - Very Good Results
it really cut by half loading times, on top of our optimisation code i think we are on track
We will be launching tomorrow the first package with performance fixes, we did several things but perhaps where you will notice now immediate results is in any section (policy, control, etc) that has custom fields, in particular many custom fields.
Our test was done on the policy section with 100 policies and 260 custom fields (we dont think anyone has this many custom fields but…)
loading times on 2.0.11 (current) and 2.0.12 (due for tomorrow) can be summarised here:
There is some catching enabled so loading on the second time (when no modifications have been made) will typically be faster than the first load.
In 10 days (end of de month) the second run of optimisations will be ready, aiming at pagination. This will mean that you will be able to have 5000 items in one section and the system wont stall. We have also tested FPM instead of mod_php with very good results, but we still need to test it further. We have been lucky to engage a really good guy (one of the core coders of Squid since late 90’s) to help us think about it.
we are completing a release today or next monday, among other things we have improved how caching works in eramba. the first load of a section is slow , subsequent loads are quicker because content (part of it) is stored on files on a cache (this saves many small disk-annoying queries).
every time you add, edit or delete an item on the section, the FULL cache gets reseted for that section … so on the next load ? slow again. the fix we made makes only the edited object to be flushed from the cache, not the entire section. this makes subsequent loads go faster.
we will be doing some measurements toady / friday to see how much faster it goes, we’ll publish results here. we have also explored memcache instead of disk cache, likekely to become a feature in coming releases … but since cached content is small (under 10 mg) we thing there wont be a huge deal of improvements.
Does this also solve the performance issue regarding exempted user group visualisation disabling/enabling?
Maybe, the obvious speed improvement will be when:
we will work in the coming releases on the loading time (index) that as you can see on the “first load” (no cache) takes minutes to load , subsequent loads as you can see are a few seconds. The gain here is that the system is usable when you have thousands of items, before you had to wait minutes and minutes for every add, edit or delete.
