Our Software, Release, SaaS Infrastructure Explained

Forum - Software
1

This post is closed for comments, it explains in as much detail as we can how our Software Process, Packaging Process and SaaS service works.

Roadmap, Issue and Development Tasks:

  • We build our software roadmap with ideas that come from the community and from eramba. These ideas go to Github as Issues documented in very specific ways in order to address: functionality, testing cases and if applicable security concerns.

  • We define a milestone (release) by selecting issues based on our priorities and bugs relevance. The milestone is prepared by checking that every issue is well documented and we plan on how long it will take us to complete the development. This milestone definition is agreed in between developers, release manager and director.

  • Developers will work on their assigned issues by creating a unique branch from the latest release.

  • Once the development for any given issue is completed a PR is created

  • Based on industry best practices, secure coding principles will be applied during software development to ensure that the code is free from vulnerabilities and meets the organisation’s security requirements. The following principles are considered:

    • OWASP Top 10: OWASP (Open Web Application Security Project) Top 10 is a widely recognised document that provides a list of the top ten most critical web application security risks. Adhering to the OWASP Top 10 helps developers identify and mitigate common vulnerabilities such as injection attacks, cross-site scripting (XSS), and insecure direct object references.
    • CakePHP Security: CakePHP is a popular PHP framework that includes built-in security features and follows secure coding practices. By adhering to CakePHP Security principles, developers can leverage the framework’s security mechanisms, such as input validation, CSRF (Cross-Site Request Forgery) protection, and secure session handling, to enhance the application’s overall security.
    • PSR-12 Standard: PSR-12 (PHP Standards Recommendation 12) is a coding style guide for PHP. It provides a set of coding standards and best practices to improve code readability and maintainability. Following the PSR-12 Standard ensures consistent coding styles across the application, making identifying and addressing potential security issues easier.
    • PHP Stan: PHP Stan is a static analysis tool for PHP that helps developers identify potential errors, type mismatches, and security vulnerabilities in their code. Using PHP Stan, developers can perform static code analysis and receive feedback on potential security weaknesses, enabling them to address those issues proactively.

Automated CI for Docker Images:

  • For every Issue/PR there is a unique branch which automatically creates trough CI a ready to use Docker image
  • This automated process test: functional testing (cypress test), php unit testing (crontabs), quality code testing (syntax, etc) and a Docker Image that is also tested by launching containers and correctly checking they start.
  • The images are pushed to AWS ECR (registry)

Testing Phase:

  • Manual Testing is done for every Issue (related PR’s) using the provided Docker images. We rely on the tester experience and the issue test cases to know what needs to be tested and how.
  • There should already be some kind of evidence from the developer that shows basic testing was done.
  • If everything works correctly, the tester provides evidence, too. (screenshots or comments)
  • If testing fails, the PR is not approved and goes back to the developers.

New Package is Built:

  • When all PRs have been approved developers will merge all into a temporal branch
  • A release (as pre-release, not production) is created by developers in Github which triggers a new docker image (with all the tests we mentioned before) with the planned released issues
  • A matching package is also created (as development, not production) by developers in the Support application. This package is a ZIP file that will be used by our on-premise customers.

Now we test the package:

  • our testers (not developers) use a testing environment (test.eramba.org) to test the package. We upgrade this system with the newly created package simulating what our end customers would do.
  • our testers (not developers) leave the instance running for at least one night to ensure the night crontab process runs correctly.
  • developers run CRUD cypress tests.

Now we test the docker image simulating a fresh install:

  • our testers (not developers) deploy the docker image on their laptops to simulate a fresh install of the eramba community using this latest release.
  • our testers (not developers) deploy the docker image on their laptops to simulate a fresh install of eramba enterprise using this latest release.

Now we test the docker image on our SaaS premises:

  • our testers (not developers) will test the new docker image by upgrading test-e.cloud.eramba.org
  • our testers (not developers) will test the new docker image by deploying a new temporary tenant tmp-test-e.cloud.eramba.org
  • our testers (not developers) leave these instances running for at least one night to ensure the night crontab process runs correctly.

We test the systems for vulnerabilities:

  • our testers will run a vulnerability scanner (using application layer scans) against the enterprise on-premise test installation (acunetix.eramba.org)
  • our testers will run a vulnerability scanner (using infrastructure scans, not application) against the enterprise on-premise test installation (tmp-test-e.cloud.eramba.org)

Now we can publish the release for on-premise customers:

  • Our testers create a forum post that includes all issues we worked on and copies of the acunetix scans
  • Our testers switch the release from development to production on support application, this makes the release visible to all our customers installs
  • our testers send an email to all our customers notifying them about the new release

Now we can publish the release for SaaS customers:

  • Our testers will migrate saas customers in batches by manually changing their stats on our licenses software. the automation takes care of the deployment of the new docker image to the tenants.

SaaS Technical Standards:

  • Eramba owns an AWS organization that is made of multiple accounts. This is to ensure segregation of environments and access permissions.
  • Login to AWS using the console (very unlikely to be used) is done over SSO and MFA.
  • We use AWS US and EU regions for production (where our tenants sit) and for development we use the EU alone.
  • There are no “clicks” done on AWS web interface, the entire infrastructure is built with Infrastructure as Code (IaC) technologies. Infrastructure code is written in Terraform and stored in our Github account.
  • This is handled by a trusted third party company (https://lablabs.io/) with whom we have long term contracts in place. LabLabs has limited and audited access to our AWS accounts, Infrastructure and Github.
  • Any change on the code is managed and approved (in between LabLab consultants and eramba) using Github PRs and approvals.
  • We test changes in our AWS development account before pushing them into Production accounts.
  • Eramba needs the following services to run: Apache, MySQL, Caching and a file system where to store your attachments. In AWS these services run as follows:
    • MySQL runs on a managed AWS RDS Aurora service. Every customer has its own logical database and its own credentials and grants. Data there is encrypted using AWS KMS. Daily backups are done and stored for up to 30 days.
    • Caching is done on AWS ElasticCache, “Redis”. Each customer has its own logical cache, with its own separated space. This is also encrypted with AWS KMS.
    • Docker container which contains Apache and eramba application is running as a deployment with two replicas on an EKS cluster.
    • Attachments customer upload are stored in AWS EFS. Each tenant has a logical separation by dedicated access points. The filesystem is encrypted with AWS KMS. Daily backups are done and stored for up to 35 days.
    • Each region we run AWS Application Load Balancers with AWS Web Application Firewall (Web Access Control Lists). We use TLS certificates managed by AWS ACM.
  • The way we provision, deprovision and upgrade customers is by using automated Kubernetes Deployments procedures running on ArgoCD.
  • When a customer is deprovisioned ArgoCD automatically runs a deployment to remove all tenant related services. Database data is removed permanently (with the exception of backups) and uploaded documents (with the exception of backups).

Information security is integrated into project management to ensure security risks are identified, assessed, and managed throughout the project lifecycle. This includes:

  • Considering security requirements during project planning and design
  • Incorporating security controls into the eramba application
  • Conducting regular security assessments and monitoring during project execution

The eramba development team works closely with the information security team to ensure that security risks are properly identified and mitigated. The team follows the information security policies and procedures established by the organisation, and any deviations from these policies should be approved by the information security team.

Monitoring:

  • We rely on a mix of: AWS CloudWatch (to collect all AWS service metrics), Prometheus and Loki (to collect eramba application logs). It all is reported using Grafana.
  • We have specific alarms for specific known risks (we can not make public) defined in Grafana Alerting. These alarms reach eramba teams in multiple ways (we can not make this public).
  • You can review our uptime at https://uptime.eramba.org
1 Like