Infrastructure

Cloud

As a rule, Curation uses cloud SAAS solutions, unless they:

• are prohibitively expensive
• would compromise the security of our data/our clients' data
• cannot effectively or efficiently perform the same role as an on-prem solution.
• could put sales opportunities a risk

We primarily use AWS, though development instances do exist on both Google Cloud and Azure. We aim to be platform-agnostic between cloud providers, though this is tempered by the obvious provider incentive for vendor lock-in.

We will provide a console login and necessary creds scoped for your role. You will also be given a walkthrough on any services relevant to your work, but further training is available should you want it.

Kubernetes

We aim to run all of our internet-facing applications on a single Kubernetes (k8s) cluster, for the following reasons:

• fault tolerance
• scalability - both horizontal and vertical autoscaling of pods and nodes according to demand
• CI/CD integration

K8s is namespaced at a project level, eg. the website resides in the website namespace. Where possible, connect K8s to your repository and use Gitlab Autodevops to deploy your services, rather than having to maintain another CI configuration.

K8s is configured to use EC2 spot instances for non-production deployments. This affords us with more compute capacity, with a trade-off that AWS may intermittently requisition our instances, resulting in temporary downtime. K8s handles this pretty well, so any downtime should be fairly limited (on the order of single digit seconds), depending on the state of k8s resource requests and the availability of spot instances.

Gitlab

We exclusively use Gitlab as our repository and CI/CD solution. Developers should ensure repositories are connected to Kubernetes cluster to enable service deployment, and to utilise the on-cluster Gitlab Runner.

Datastores

Our primary datastore vendor is AWS, and we have instances of the following available:

• RDS (postgres/SQL Server)
• Amazon Elasticsearch Service
• Elasticsearch cloud
• Elasticache

Development databases are accessible by all developers through shared logins available in the password manager. Production databases are restricted access. You may be given read-only access to production databases, and in exceptional circumstances write-access, but these will be time-limited, supervised and queries logged.

All database ingress is VPC-only. Under no circumstances will any database instance accept internet ingress, even during development. If you need to access from outside the office, use the Client VPN.

Serverless solutions are preferred over managed ones. Managed solutions are preferred over hosted. Relational databases are preferred over KV stores. KV stores are preferred over document-oriented datastores, but this is all dependent on your data. Data are usually more relational than they may appear. Data modelling shortcuts taken at the beginning of a project frequently cost time down the line. Always consider if you need a new instance versus a new datastore.