DevOps et Site Reliability Engineering (SRE)
Le DevOps s'inscrit dans la culture d'entreprise comme un ensemble de principes qu'une entreprise aspire et respecte à long terme. Les partisans de cette culture valorisent la collaboration, l’expérimentation et la volonté d’apprendre. Toutes les parties impliquées dans une culture DevOps se concentrent sur un objectif tout au long du cycle de vie d'un projet (pas seulement le développement et l'exploitation): la mise en œuvre rapide de logiciels stables et de haute qualité, du concept client jusqu'à l'expérience utilisateur.
L'automatisation du développement, des tests et du déploiement de logiciels par le biais du déploiement continue (CD) est un facteur clé reconnu de la méthodologie DevOps. L'automatisation permet une mise en œuvre logicielle plus rapide et garantit à la solution finale qualité, sécurité et stabilité.
Objectifs
Définition et contribution:
- Indicateur de niveau de service (SLI)
- Objectif de niveau de service (SLO)
- Service Level Agreements (SLA)
- Estimation des risques, des niveaux de disponibilité et du budget d'erreur
Collaboration
Expertise auprès des développeurs d'applications:
- Gestion du changement
- Définir des objectifs communs
- Assurer la livraison de la production
- Améliorer la fiabilité du système
Responsabilités
Implication et responsibilisation:
- Surveillance et notifications
- Estimation des capacités allouées
- Latence, performance et efficacité
- Réponse d'urgence et automatisation
Articles associés au DevOps
Local development environments with Terraform + LXD
Catégories : Containers Orchestration, DevOps & SRE | Tags : Automation, DevOps, KVM, LXD, Virtualization, VM, Terraform, Vagrant
As a Big Data Solutions Architect and InfraOps, I need development environments to install and test software. They have to be configurable, flexible, and performant. Working with distributed systems…
Par Gauthier LEONARD
1 juin 2023
How to build your OCI images using Buildpacks
Catégories : Containers Orchestration, DevOps & SRE | Tags : CNCF, OCI, CI/CD, Docker, Kubernetes
Docker has become the new standard for building your application. In a Docker image we place our source code, its dependencies, some configurations and our application is almost ready to be deployed…
9 janv. 2023
Big data infrastructure internship
Catégories : Big Data, Data Engineering, DevOps & SRE, Infrastructure | Tags : Infrastructure, Hadoop, Big Data, Cluster, Internship, Kubernetes, TDP
Job description Big Data and distributed computing are at the core of Adaltas. We accompagny our partners in the deployment, maintenance, and optimization of some of the largest clusters in France…
Par Stephan BAUM
2 déc. 2022
Introducing Trunk Data Platform: the Open-Source Big Data Distribution Curated by TOSIT
Catégories : Big Data, DevOps & SRE, Infrastructure | Tags : DevOps, Hortonworks, Ansible, Hadoop, HBase, Knox, Ranger, Spark, Cloudera, CDP, CDH, Open source, TDP
Ever since Cloudera and Hortonworks merged, the choice of commercial Hadoop distributions for on-prem workloads essentially boils down to CDP Private Cloud. CDP can be seen as the “best of both worlds…
Par Leo SCHOUKROUN
14 avr. 2022
JS monorepos in prod 7: Continuous Integration and Continuous Deployment with GitHub Actions
Catégories : DevOps & SRE, Front End | Tags : CI/CD, Monorepo, Node.js, Unit tests
The value of CI/CD lies in the ability to control and coordinate changes and feature addition in multiple, iterative releases while simultaneously having multiple services being actively developed in…
6 avr. 2022
Ansible variables: choosing the right location
Catégories : DevOps & SRE | Tags : Infrastructure, Ansible, IaC, YAML
Defining variables for your Ansible playbooks and roles can become challenging as your project grows. Browsing the Ansible documentation, the diversity of Ansible variables location is confusing, to…
Par Xavier HERMAND
15 mars 2022
GitOps in practice, deploy Kubernetes applications with ArgoCD
Catégories : Containers Orchestration, DevOps & SRE, Adaltas Summit 2021 | Tags : Argo CD, CI/CD, Git, GitOps, IaC, Kubernetes
GitOps is a set of practices to deploy applications using Git. Application definitions, configurations, and connectivity are to be stored in a version control software such as Git. Git then serves as…
16 déc. 2021
JS monorepos in prod 6: CI/CD, continuous integration and deployment with Travis CI
Catégories : DevOps & SRE, Front End | Tags : CI/CD, Monorepo, Node.js, Unit tests
Implementing continuous integration CI and continuous deployment (CD) on a monorepo is quite complex due to the diversity of multiple responsibilities between developers and the need to coordinate…
Par David WORMS
6 déc. 2021
Running your Travis CI builds locally with Docker
Catégories : DevOps & SRE, Front End | Tags : Bash, Tools, CI/CD, Node.js, Unit tests
Setting up the environment to run the tests on a CI/CD can take a few roundtrips between your host machine and the CI/CD running remotely. For every attempt, you’ll have to commit and publish your…
Par David WORMS
6 sept. 2021
Modern Python part 3: run a CI pipeline & publish your package to PiPy
Catégories : DevOps & SRE | Tags : CI/CD, Git, GitHub, Python, Release and features, Unit tests
To propose a well-maintained and usable Python package to the open-source community or even inside your company, you are expected to accomplish a set of critical steps. First ensure that your code is…
Par Faouzi BRAZA
28 juin 2021
Modern Python part 2: write unit tests & enforce Git commit conventions
Catégories : DevOps & SRE | Tags : Git, pandas, Python, Unit tests
Good software engineering practices always bring a lot of long-term benefits. For example, writing unit tests permits you to maintain large codebases and ensures that a specific piece of your code…
Par Faouzi BRAZA
24 juin 2021
Modern Python part 1: start a project with pyenv & poetry
Catégories : DevOps & SRE | Tags : Git, Python, Release and features, Unit tests
When learning a programming language, the focus is essentially on understanding the syntax, the code style, and the underlying concepts. With time, you become sufficiently comfortable with the…
Par Faouzi BRAZA
9 juin 2021
JS monorepos in prod 5: merging Git repositories and preserve commit history
Catégories : DevOps & SRE, Node.js | Tags : Bash, DevOps, Packaging, Git, GitHub, GitOps, JavaScript, Monorepo
At Adaltas, we maintain several open-source Node.js projects organized as Git monorepos and published on NPM. We shared our experience to work with Lerna monorepos in a set of articles: Part…
Par Sergei KUDINOV
21 mai 2021
Bridging the DBnomics Swagger/OpenAPI schema with GraphQL
Catégories : DevOps & SRE, Front End | Tags : Data Engineering, JAMstack, GraphQL, JavaScript, Node.js, REST, Schema
While redacting a long and fastidious document today, I came across DBnomics, an open platform federating economic datasets. Browsing its website and APIs, I found their OpenAPI schema (aka Swagger…
Par David WORMS
8 avr. 2021
JS monorepos in prod 4: unit testing with Mocha and Should.js
Catégories : DevOps & SRE, Front End | Tags : Automation, CI/CD, Git, GitOps, Monorepo, Node.js, Unit tests
Unit testing is essential for every long-term project and allows you to pull down functionalities of your code into isolated testable units. Indeed the main goal of a unit test is to verify if an …
Par David WORMS
25 févr. 2021
JS monorepos in prod 3: commit enforcement and changelog generation
Catégories : DevOps & SRE, Front End | Tags : CI/CD, Git, JavaScript, Monorepo, Node.js, Release and features, Unit tests
Conventional Commits introduces a structured format for commit messages. It standardizes the messages among all the contributors. This makes them more readable and easy to automate. It simplifies the…
Par David WORMS
2 févr. 2021
JS monorepos in prod 2: project versioning and publishing
Catégories : DevOps & SRE, Front End | Tags : CI/CD, Git, GitOps, JavaScript, Monorepo, Node.js, Release and features, Unit tests
One great advantage of a monorepo is to maintain coherent versions between packages and to automatize the version creation and the publication of packages. This article covers the versioning and…
Par David WORMS
11 janv. 2021
JS monorepos in prod 1: project initialization
Catégories : DevOps & SRE, Front End | Tags : Git, GitOps, JavaScript, Monorepo, Node.js, Release and features
Every project journey begins with the step of initialization. When your overall project is composed of multiple projects, it is tempting to create one Git repository per project. In Node.js, a project…
Par David WORMS
5 janv. 2021
Data versioning and reproducible ML with DVC and MLflow
Catégories : Data Science, DevOps & SRE, Events | Tags : Data Engineering, Databricks, Delta Lake, Git, Machine Learning, MLflow, Storage
Our talk on data versioning and reproducible Machine Learning proposed to the Data + AI Summit (formerly known as Spark+AI) is accepted. The summit will take place online the 17-19th November…
30 sept. 2020
Version your datasets with Data Version Control (DVC) and Git
Catégories : Data Science, DevOps & SRE | Tags : DevOps, Infrastructure, Operation, Git, GitOps, SCM
Using a Version Control System such as Git for source code is a good practice and an industry standard. Considering that projects focus more and more on data, shouldn’t we have a similar approach such…
Par Grégor JOUET
3 sept. 2020
Machine Learning model deployment
Catégories : Big Data, Data Engineering, Data Science, DevOps & SRE | Tags : DevOps, Operation, AI, Cloud, Machine Learning, MLOps, On-premises, Schema
“Enterprise Machine Learning requires looking at the big picture […] from a data engineering and a data platform perspective,” lectured Justin Norman during the talk on the deployment of Machine…
Par Oskar RYNKIEWICZ
30 sept. 2019
Rook with Ceph doesn't provision my Persistent Volume Claims!
Catégories : DevOps & SRE | Tags : PVC, Linux, Rook, Ubuntu, Ceph, Cluster, Internship, Kubernetes
Ceph installation inside Kubernetes can be provisioned using Rook. Currently doing an internship at Adaltas, I was in charge of participating in the setup of a Kubernetes (k8s) cluster. To avoid…
Par Eyal CHOJNOWSKI
9 sept. 2019
Spark Streaming part 3: DevOps, tools and tests for Spark applications
Catégories : Big Data, Data Engineering, DevOps & SRE | Tags : DevOps, Learning and tutorial, Spark, Apache Spark Streaming
Whenever services are unavailable, businesses experience large financial losses. Spark Streaming applications can break, like any other software application. A streaming application operates on data…
Par Oskar RYNKIEWICZ
31 mai 2019
Monitoring a production Hadoop cluster with Kubernetes
Catégories : DevOps & SRE | Tags : Thrift, Grafana, Shinken, Hadoop, Knox, Cluster, Docker, Elasticsearch, Kubernetes, Node, Node.js, Prometheus, Python
Monitoring a production grade Hadoop cluster is a real challenge and needs to be constantly evolving. The software we use today is based on Nagios. Very efficient when it comes to the simplest…
21 déc. 2018
Hadoop cluster takeover with Apache Ambari
Catégories : Big Data, DevOps & SRE, Adaltas Summit 2018 | Tags : Ambari, Automation, iptables, Kerberos, Nikita, Systemd, Cluster, HDP, Node, Node.js, REST
We recently migrated a large production Hadoop cluster from a “manual” automated install to Apache Ambari, we called this the Ambari Takeover. This is a risky process and we will detail why this…
Par Leo SCHOUKROUN
15 nov. 2018
KVM machines for Vagrant on Archlinux
Catégories : DevOps & SRE | Tags : Arch Linux, KVM, Linux, Virtualization, VM, Vagrant
Vagrant supports different providers to manage virtualization. In a Linux environment, you can dramatically improve VM performance by using the libvirt provider and the KVM hypervisor. This tutorial…
Par Gauthier LEONARD
19 sept. 2018
Publishing guidelines
Catégories : DevOps & SRE | Tags : Arch Linux, KVM, VM, Vagrant, Markdown
This is as much a set of guidelines targeting everyone publishing content on the web as rules for reviewers to ensure no validation is forgotten before submitting for publication. It mostly targets…
Par David WORMS
28 févr. 2018
Ambari - How to blueprint
Catégories : Big Data, DevOps & SRE | Tags : Ambari, Automation, DevOps, Operation, Ranger, REST
As infrastructure engineers at Adaltas, we deploy Hadoop clusters. A lot of them. Let’s see how to automate this process with REST requests. While really handy for deploying one or two clusters, the…
Par Joris RUMMENS
17 janv. 2018
Multi-Repo, Multi-Node Gating at Massive Scale
Catégories : Cloud Computing, DevOps & SRE, Open Source Summit Europe 2017 | Tags : Infrastructure, Jenkins, Red Hat, Zuul, Ansible, CI/CD, OpenStack
This is a recap and personal review of Monty Taylor’s presentation of OpenStack’s Continuous Integration tool Zuul at the OpenSource Summit 2017 in Prague (not to mix with Netflix’ Zuul project…
Par Joris RUMMENS
28 oct. 2017
Apache Thrift vs REST
Catégories : DevOps & SRE, Open Source Summit Europe 2017 | Tags : Thrift, gRPC, HTTP, JavaScript Object Notation (JSON), REST
Adaltas recently attended the Open Source Summit Europe 2017 in Prague. I had the opportunity to follow a presentation made by Randy Abernethy and Jens Geyer of RM-X, a cloud native consulting company…
Par Leo SCHOUKROUN
28 oct. 2017
From Dockerfile to Ansible Containers
Catégories : Containers Orchestration, DevOps & SRE, Open Source Summit Europe 2017 | Tags : pip, Shell, Ansible, Docker, Docker Compose, YAML
This talk was an introduction to the Dockerfile format and to Ansible container’s tool and then a comparison of both. It was hold by Tomas Tomecek from Red Hat’s containerization team. The Dockerfile…
Par César BEREZOWSKI
25 oct. 2017
MiNiFi: Data at Scales & the Values of Starting Small
Catégories : Big Data, DevOps & SRE, Infrastructure | Tags : MiNiFi, NiFi, C++, HDF, Cloudera, HDP, IOT
This conference presented rapidly Apache NiFi and explained where MiNiFi came from: basically it’s a NiFi minimal agent to deploy on small devices to bring data to a cluster’s NiFi pipeline (ex: IoT…
Par César BEREZOWSKI
8 juil. 2017
HDP cluster monitoring
Catégories : Big Data, DevOps & SRE, Infrastructure | Tags : Alert, Ambari, Metrics, Monitoring, HDP, REST
With the current growth of BigData technologies, more and more companies are building their own clusters in hope to get some value of their data. One main concern while building these infrastructures…
Par Joris RUMMENS
5 juil. 2017
Hive Metastore HA with DBTokenStore: Failed to initialize master key
Catégories : Big Data, DevOps & SRE | Tags : Infrastructure, Hive, Bug
This article describes my little adventure around a startup error with the Hive Metastore. It shall be reproducable with any secure installation, meaning with Kerberos, with high availability enabled…
Par David WORMS
21 juil. 2016
A fresh look at testing Node.js projects: Mocha, Should and Travis
Catégories : DevOps & SRE, Node.js | Tags : DevOps, Mocha, CI/CD, JavaScript, Node.js, Unit tests
Today, I finally decided to spend some time around Travis. It’s been a few weeks since that little green image on top of many GitHub homepages has been buzzing me. Well, to be totally honest, this isn…
Par David WORMS
19 févr. 2012
Announcing Mecano, a set of functions for system deployment
Catégories : DevOps & SRE, Node.js | Tags : Automation, Infrastructure, CoffeeScript, JavaScript, Open source
Update July 2016, Mecano is now renamed Nikita. We are releasing Node Mecano on GitHub which gather common functions used while deploying systems. The idea was to group those functions into a…
Par David WORMS
12 févr. 2012
