dev_factory_devops - cool repo for building production like setups consisting of many hosts, database, servers, etc.
- Ubuntu 20.04
- One head (more or less clean)
- Few hands sticking out of proper place
Sooner or later some developers working on a complex application setup will want to have a look how his solution works in production-like envoronment.
A complex app setup might consist of many hosts - some running appications, some running databases and in memory key-value storages, web servers any many others.
In this case developer will find that there is a bunch of tools like Vagrant, Ansible and many others for producing this kind of environments.
This repository is meant to create this kind of environments, to learn all of these tools and have a look how it will work.
To prove that everything mentioned below works here is test environment setup is prepared.
You can simply navigate to that example and try it out.
Finishing reading is recommened though.
tsrc/example1
Simple environment example below consists of 3 VMs:
- h1
- dns server
- h2
- database
- h3
- server
Tough environment example below is much closer to real case:
- h1
- dns server
- h2
- database
- metrics exporter
- agent (docker proxy for Portainer)
- h3
- auth server
- metrics exporter
- agent (docker proxy for Portainer)
- h4
- memcached for storing sessions
- metrics exporter
- agent (docker proxy for Portainer)
- h5
- app server
- metrics exporter
- agent (docker proxy for Portainer)
- h6
- Portainer - docker managment tool (web server)
- Prometheus - metrics database (web server)
- Grafana - metrics visualization tool (web server)
metrics exporter in this case is a simple http server which reports correspondng metrics to Prometheus
agent in this case is a docker engine proxy ustilized by Portainer
Vagrant is a tool for building and maintaining virtual machines using many providers like VirtualBox, KVM, VMware, and others.
Vagrant can be considered as a wrapper for VirtualBox or any other provider.
Virtual machines are desribed by a Vagrantfile which is actually Ruby file.
$cat Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
servers=[
{
:hostname => "h1",
:ip => "192.168.200.100",
:box => "generic/ubuntu2004",
:ram => 2048,
:cpu => 1,
:ports => [
{
:host => 3000,
:guest => 3000,
},
]
},
{
:hostname => "h2",
:ip => "192.168.200.101",
:box => "generic/ubuntu2004",
:ram => 1024,
:cpu => 1,
:ports => [
{
:host => 4000,
:guest => 3000,
},
]
}
]
Vagrant.configure("2") do |config|
servers.each do |machine|
config.vm.define machine[:hostname] do |node|
node.vm.box = machine[:box]
node.vm.hostname = machine[:hostname]
node.vm.network "private_network", ip: machine[:ip]
node.vm.provider "virtualbox" do |vb|
vb.customize ["modifyvm", :id, "--memory", machine[:ram]]
end
# forward ports
machine[:ports].each do |ports|
node.vm.network "forwarded_port", guest:ports[:guest], host:ports[:host]
end
end
end
end
Once Vagrantfile is defined, you can
$ vagrant status
Current machine states:
h1 poweroff (virtualbox)
h2 poweroff (virtualbox)
This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
$ vagrant up
$ vagrant halt
Once of the baset Vagrant features is ability to make/restore VMs snapshots.
To make snapshots:
$ vagrant snapshot save v0
To restore snapshots:
$ vagrant snapshot restore v0
Installing Vagrant is a piece of cake, here is shell script which will do this for you.
$ cd env
$ sudo vagrant.install.sh
VirtualBox - a virtualizer for x86-x64 machines, i.e. for virtual machines.
We planning using Vagrant for VMs managemtnt - so VirtualBox is not meant to be used directly.
Still, VirtualBox must be installed.
Once VirtualBox is installed you can create/delete/view virtual machines using VirtualBox UI tool.
VirtualBox can be installed right from official Ubuntu repositories.
$ sudo apt update
$ sudo apt install virtualbox
Ansible - a devops automation tool.
If briefly - Ansible helps to run playbooks(recipes) on remote hosts tolet say install a package, create a file, etc.
Once playbook is run Ansible:
- Reads yaml playbooks and executes ones to every listed host(VM).
- Uses ssh to connect to the hosts. It means ssh public key shall be delivered to every VM.
$ ssh-copy-id -i ~/.ssh/id_rsa.pub vagrant@h1
- Has a lot of modules which can be run on a remote host and perform things like install packages, launch shell commands and many others:
$ ansible-playbook -i hosts -K main.yml
Installing Ansible is another piece of cake, here is shell script which will do this for you.
$ cd env
$ sudo ansible.install.sh
Portainer is powerful tool for managing docker engine, i.e. containers, images, networks, stacks, etc..
Consists of 2 parts:
- Portainer itself, web server providing UI for managing docker engine - launching containers, and everything what docker can do
- Portainer agent - a proxy for ruling docker-engines located at another hosts
Portainer is not meant for installation on main(working) host directly.
Portainer is meant to be installed on a single host conforming the environment.
Portainer agent is meant to be installed on every host(excluding host with Portainer) conforming the environment.\
Down here are some exampleis what Portainer looks like.
Prometheus is monitoring system & time series database.
Collects(polls) metrics from remote hosts over http.
Most tools like postgress, docker and others have exporters for Prometheus.
Has a web server(Frontend) for presenting metrics and other stuff.
Prometheus is not meant for installation on main(working) host directly.
Prometheus is meant to be installed on a single host conforming the environment.
Using dockprom for complete Prometheus + Grafana setup.
Down here is an example what Prometheus charts look like.
Grafana is metrics visualization tool.
Collects and visualizes metrics from Prometheus.\
Presenting mestrics via a set of Dashboards where each Dashboard has several charts.
It's a web server(Frontend) presenting Prometheus data using nice charts.
Grafana is not meant for installation on main(working) host directly.
Grafana is meant to be installed on a single host conforming the environment.
Using dockprom for complete Prometheus + Grafana setup.
Down here is an example what Grafana charts look like.
docker and Co is well know set of tools for managing docker images, containers, running applications in swarm mode(many workers with load balancing)