Deployment

This guide is meant to give a high-level overview of deployment techniques and tips when planning how to deploy InvenioILS.

Read the infrastructure architecture first!

This section assumes you have a good understanding of the different services that are needed to run an InvenioILS application, and how they interact with each other. This information can be found in the infrastructure architecture.

Deployment models

There are many aspects to take into account: your knowledge in DevOps/Platform engineering, your organization's infrastructure, resources, constraints or policies, the technology you want to invest in and maintain, pricing, etc.

The mentioned technologies are just examples

Many technologies will be named. However, note that those are mere examples based on our experience at CERN or from some of the Invenio partners, with all certainty there are many other alternatives for each case.

Two popular deployment models are:

• Platform as a Service (PaaS): using this model you will be in charge of the application and the data it handles. This more commonly known as containers, meaning that you will be in charge of packaging your instance's InvenioILS application in a container image. For example, using the Dockerfile that is created when you bootstrapped your instance. At CERN we deploy our instances on OpenShift.

• Infrastructure as a Service (IaaS): using this model you will be in charge of the stack from the operative system up. This is more commonly known as virtual machines, meaning that you will be in charge of configuring the operative system, Python, the web server, etc. At CERN we have been running Invenio v3 instances using OpenStack to manage the VMs, and Puppet for IaC (Infrastructure as Code). Other common IaC technologies are Ansible and Terraform.

You could also keep going down in the layers to get more and more control of your deployment, for example, using bare metal machines. However, that is out of the scope of this documentation.

Docker compose is not recommended

The provided docker-compose files in InvenioILS can give you a good idea of what you need to deploy, including the underlying services.

However, such configuration is NOT meant to be used in production as-is: it requires knowledge and experience on how to run it reliably, safely stores files with backups, managing software and infrastructure upgrades, understand potential performance limitations (a single machine probably might not handle all your traffic and it would be a single point of failure), etc.

Use at your own risk.

Services

When deploying InvenioILS, you can choose to install, securely configure and maintain yourself the services (such as the database, the search engine, etc.) services, or use third party providers. For example, the three cloud providers mentioned above (AWS, GC, and Azure) can provide most of the services. If you choose to deploy them yourself you will need to deep-dive and get experienced in the following topics:

• Persist your data, and enable periodic backups. This includes the relational database, the search indices and the files.
• Queue persistence (RabbitMQ) to avoid losing tasks in case the service fails.
• High availability, many of the services can be deployed redundantly. Note that most of the cloud providers offer this option or have an established SLA.
• Secrets handling. Most of the services require credentials to connect. It is a good practice to keep them in a secure place, for example Ansible Vault.
• Security. This is a broad topic, and one of the most important. It is also highly dependent on your deployment model. Take into account exposing only the minimal amount of services needed and safeguard the access to your infrastructure.

If you choose to deploy them using PaaS, images for the services can be found for example in DockerHub. For more information on how to deploy each service, refer to their official documentation.

InvenioILS Python app

Independently of your deployment model choice, you will need to run the Flask Python application. However, the server bundled with Flask is not meant for production usage, it is there only to help with the development. Therefore, you will need some sort of WSGI HTTP server.

Common choices are Gunicorn and uWSGI. The latter, is the one we use at CERN and it is bundled in the Dockerfile.

Alerting

Once the monitoring is in place, the next step is getting alerts for code exceptions and when certain metrics reach a configured threshold.

For the InvenioILS Python application, Sentry provides a great level of details (variables' values, stacktraces, etc.) when exceptions happen: this is extremely helpful when it comes to understanding errors that happen in production. If you have to invest in one alerting tool, we really recommend this one!

In the Grafana ecosystem, OnCall. Alternatively, many messaging apps can be configured with webhooks to Grafana.

Security

To secure your instance, make sure that:

• You have correctly configured your InvenioILS instance.
• You have valid HTTPS certificates.
• Your instance and services are up-to-date.

This list is not meant to be exhaustive: please refer to well-known literature and best practices on how to make your infrastructure secure.

Load testing

It might be interesting to understand what is the load that your system will sustain, to size your infrastructure accordingly. That can be tested for example with locust or k6.

Troubleshooting

The first step to troubleshooting a problem would be to try to understand in which layer or service of the system is the problem. That can be checked using the monitoring and dashboard technologies mentioned above, or the log files directly.

If the problem is in the InvenioILS Python application, you can get more details in Sentry. Afterwards, you can try to reproduce it and debug in a local installation.