This post is closed for comments, it explains in as much detail as we can how our SaaS service works.
It all starts when we develop ideas into software:
- We build our software roadmap with ideas that come from the community and from eramba. These ideas go to Github as Issues documented in very specific ways in order to address usability, security and functional needs.
- Our software developers have been with us from the beginning and know the software very well, they have received formal training in Web Application Security.
- On every Issue a Github Pull Request (PR) is created and tested by a separate team. If the issue complies with the specifications (security, usability and functionality) the PR is approved and can be merged to the Github Branch created for this release (we use tags, Etc).
- On every open PR, our Continuous Integration (CI) pipeline executes multiple checks: Coding Standards, Static Analysis, Functional Testing, Creation of Docker Images, Testing of Docker Images, Pushing Images to AWS ECR, Etc.
- At this point we already have an image we can use to do manual testing deploying these images on our laptops and AWS development environment. If the testing goes well we approve the PR, if not it goes back to development.
- On-Premise (not SaaS) customers use TAR packages to update their instances, these TAR packages are created by our support software on demand. We use them to test upgrades that require migration of data.
- All testing is done using dummy databases with fictitious data. No customer or real data is ever used.
- Once all PRs on the planned release have been approved they are all merged into a single branch which again triggers the CI pipeline creating a final image with all issues (patches and features) in it.
- We run a vulnerability scanner (typically Acunetix, sometimes PortSwigger too) against the image and ensure no issues are found.
- We can now simulate upgrading both On-Premise and SaaS development environments to ensure the upgrade process will work on end customers.
At this point we have new software already packaged and tested ready for releases. We will now briefly explain how our SaaS infrastructure is set up.
- Eramba owns an AWS organization that is made of multiple accounts. This is to ensure segregation of environments and access permissions.
- Login to AWS using the console (very unlikely to be used) is done over SSO and MFA.
- We use AWS US and EU regions for production (where our tenants sit) and for development we use the EU alone.
- There are no “clicks” done on AWS web interface, the entire infrastructure is built with Infrastructure as Code (IaC) technologies. Infrastructure code is written in Terraform and stored in our Github account.
- This is handled by a trusted third party company (https://lablabs.io/) with whom we have long term contracts in place. LabLabs has limited and audited access to our AWS accounts, Infrastructure and Github.
- Any change on the code is managed and approved (in between LabLab consultants and eramba) using Github PRs and approvals.
- We test changes in our AWS development account before pushing them into Production accounts.
- Eramba needs the following services to run: Apache, MySQL, Caching and a file system where to store your attachments. In AWS these services run as follows:
- MySQL runs on a managed AWS RDS Aurora service. Every customer has its own logical database and its own credentials and grants. Data there is encrypted using AWS KMS. Daily backups are done and stored for up to 30 days.
- Caching is done on AWS ElasticCache, “Redis”. Each customer has its own logical cache, with its own separated space. This is also encrypted with AWS KMS.
- Docker container which contains Apache and eramba application is running as a deployment with two replicas on an EKS cluster.
- Attachments customer upload are stored in AWS EFS. Each tenant has a logical separation by dedicated access points. The filesystem is encrypted with AWS KMS. Daily backups are done and stored for up to 35 days.
- Each region we run AWS Application Load Balancers with AWS Web Application Firewall (Web Access Control Lists). We use TLS certificates managed by AWS ACM.
- The way we provision, deprovision and upgrade customers is by using automated Kubernetes Deployments procedures running on ArgoCD.
- When a customer is deprovisioned ArgoCD automatically runs a deployment to remove all tenant related services. Database data is removed permanently (with the exception of backups) and uploaded documents (with the exception of backups).
While we built this very automated process to keep operations cost as low as possible, there is still a lot to monitor to make sure we meet availability and security standards.
- We rely on a mix of: AWS CloudWatch (to collect all AWS service metrics), Prometheus and Loki (to collect eramba application logs). It all is reported using Grafana.
- We have specific alarms for specific known risks (we can not make public) defined in Grafana Alerting. These alarms reach eramba teams in multiple ways (we can not make this public).
- You can review our uptime at https://uptime.eramba.org