MicroSuite

µSuite: A Benchmark Suite for Microservices

µSuite is a suite of OLDI services that are each composed of front-end, mid-tier, and leaf microservice tiers. μSuite includes four OLDI services that incorporate open-source software: a content-based high dimensional search for image similarity — HDSearch, a replication-based protocol router for scaling fault-tolerant key-value stores — Router, a service for performing set algebra on posting lists for document retrieval — Set Algebra, and a user-based item recommender system for predicting user ratings — Recommend. µSuite was originally written to evaluate OS and network overheads faced by microservices. You can find more details about µSuite in our IISWC paper (http://akshithasriraman.eecs.umich.edu/pubs/IISWC2018-%CE%BCSuite-preprint.pdf).

This µSuite Fork has been amended by ALPS in order to achieve the following:

• Correct and confirm all the installation/compilation commands to run on Ubuntu Linux 18.04+
• Provide instructions to compile and run with Podman and prepare a container image with the complete µSuite for easier deployment
• Provide instructions and the configuration to run the applications on single node using podman-compose
• Provide instructions and the configuration to run the applications on multiple nodes using podman-compose
• Provide instructions and source code to run the application on single node allowing the system to enter C6 deep sleep state
• Research resource disaggregation and container snapshotting for improved energy proportionality

Energy Proportionality Research Module

This fork includes a complete research module for investigating resource disaggregation and container snapshotting:

energy_proportionality/
├── scripts/           # Checkpoint/restore, C6 monitoring, experiment runner
├── configs/           # Docker compose with CRIU support
├── analysis/          # Python analysis tools
└── results/           # Experiment output

Documentation:

• Module README - Quick start guide and usage
• Research Overview - High-level investigation of resource disaggregation
• Implementation Guide - Technical implementation details

Quick Start:

# Setup disaggregated memory simulation
sudo ./energy_proportionality/scripts/setup_disaggregated_memory.sh

# Run experiment
./energy_proportionality/scripts/run_experiment.sh test_run 10

# Analyze results
python3 energy_proportionality/analysis/analyze_results.py energy_proportionality/results/test_run_*/

License & Copyright

µSuite is free software; you can redistribute it and/or modify it under the terms of the BSD License as published by the Open Source Initiative, revised version.

µSuite was originally written by Akshitha Sriraman at the University of Michigan, and per the the University of Michigan policy, the copyright of this original code remains with the Trustees of the University of Michigan.

If you use this software in your work, we request that you cite the µSuite paper ("μSuite: A Benchmark Suite for Microservices", Akshitha Sriraman and Thomas F. Wenisch, IEEE International Symposium on Workload Characterization, September 2018), and that you send us a citation of your work.

Installation

To install µSuite, please follow these steps (works on Ubuntu 18.04+):

(1) ** Setup Podman and Podman Compose **

sudo apt update
sudo apt -y install podman podman-compose

For saving registry login to be able to push images

sudo apt -y install gnupg2 pass

Change the storage folder for more space to commit the image (in our case when we use Cloudlab)

podman rm -f $(podman ps -aq); podman rmi -f $(podman images -q)
sudo systemctl stop podman 2>/dev/null || true
umount /var/lib/containers 2>/dev/null || true
sudo rm -rf /var/lib/containers
sudo mkdir -p /var/lib/containers
sudo mkdir -p /dev/mkpodman
sudo mount --rbind /dev/mkpodman /var/lib/containers

(2) ** Create a Podman container using our precompiled image **

mkdir microsuite
cd microsuite
git clone https://github.com/ucy-xilab/MicroSuite.git
cd MicroSuite

Run Podman Compose

podman-compose up

At this point we need to open a new terminal and login on the container to execute our benchmark

cd microsuite
podman-compose exec hdsearch sh

From this point on we can execute each benchmark based on the commands provided in section (4)

(3) ** Run a multinode execution **

All following commands should be run only on Node 0

Close any podman-compose already running through the cloudlab profile

parallel-ssh -H "node0 node1 node2" -i "cd /microsuite/MicroSuite && podman-compose down"

Download dataset

cd /microsuite/MicroSuite && sudo wget https://akshithasriraman.eecs.umich.edu/dataset/HDSearch/image_feature_vectors.dat

For multi-node Podman deployment

Note: Podman does not have a direct Swarm equivalent. For distributed deployments, consider using Kubernetes/k3s, or run podman-compose on each node separately.

export NODE0=$(ssh node0 hostname)
export NODE1=$(ssh node1 hostname)
export NODE2=$(ssh node2 hostname)

cd /microsuite/MicroSuite

# Deploy using podman-compose (runs on local node)
# For multi-node deployment, run this on each node with appropriate service configuration
podman-compose -f docker-compose-swarm.yml up -d

The provided docker-compose-swarm.yml file runs the HDSearch application. In order to run any other benchmark from the suite you need to change this file and edit the commands for each service based on the ones provided in section (4).

In addition the following commands can be used to manage and monitor the progress of the containers:

# Check running containers
podman ps

# Check logs
podman logs hdsearch_bucket
podman logs hdsearch_midtier

# Execute command in a container
podman exec -ti hdsearch_bucket bash
podman exec -ti hdsearch_midtier bash
podman exec -ti hdsearch_client bash

# Stop services
podman-compose -f docker-compose-swarm.yml down

(4) ** Run benchmarks **

** HDSearch **

Dataset for HDSearch

wget https://akshithasriraman.eecs.umich.edu/dataset/HDSearch/image_feature_vectors.dat 
mv ./image_feature_vectors.dat /home

Bucket Service Command

cd /MicroSuite/src/HDSearch/bucket_service/service
./bucket_server /home/image_feature_vectors.dat 0.0.0.0:50050 2 -1 0 1

Mid Tier Service - sudo command not found...

cd /MicroSuite/src/HDSearch/mid_tier_service/service
touch bucket_servers_IP.txt
echo "0.0.0.0:50050" > bucket_servers_IP.txt
./mid_tier_server 1 13 1 1 bucket_servers_IP.txt /home/image_feature_vectors.dat 2 0.0.0.0:50051 1 4 4 0   

Client

cd /MicroSuite/src/HDSearch/load_generator
mkdir ./results
./load_generator_open_loop /home/image_feature_vectors.dat ./results/ 1 30 100 0.0.0.0:50051 dummy1 dummy2 dummy3

** Router **

Dataset for Router

wget https://akshithasriraman.eecs.umich.edu/dataset/Router/twitter_requests_data_set.dat
wget https://akshithasriraman.eecs.umich.edu/dataset/Router/twitter_requests_data_set.txt
mv ./twitter_requests_data_set.dat /home
mv ./twitter_requests_data_set.txt /home

Memcached server - confirm it runs

service memcached restart

Lookup service command

cd /MicroSuite/src/Router/lookup_service/service
./lookup_server 0.0.0.0:50050 11211 -1 1

Mid tier

cd /MicroSuite/src/Router/mid_tier_service/service/
touch lookup_servers_IP.txt
echo "0.0.0.0:50050" > lookup_servers_IP.txt
./mid_tier_server 1 lookup_servers_IP.txt 0.0.0.0:50051 1 1 1 1

Client

cd /MicroSuite/src/Router/load_generator
mkdir ./results
./load_generator_closed_loop /home/twitter_requests_data_set.dat ./results 30 1000 0.0.0.0:50051 1 1

** SetAlgebra **

Dataset for Set algebra

wget https://akshithasriraman.eecs.umich.edu/dataset/SetAlgebra/wordIDs_mapped_to_posting_lists.txt
mv ./wordIDs_mapped_to_posting_lists.txt /home

Split dataset to multiple shards, one per intersection server

(if only one intersection_server then use whole file).

In this example we split in 10 shards (replace shards_num=10 below with number of shards you would like)

rm /home/setalgebra_shrad*.txt;shrads_num=10;split -d --additional-suffix=.txt -l $(($(($(wc -l < /home/wordIDs_mapped_to_posting_lists.txt)+shrads_num-1))/shrads_num)) /home/wordIDs_mapped_to_posting_lists.txt /home/setalgebra_shrad

Produce setalgebra_query_set.txt of N random lines (100 in this example) from dataset.

Client query set can be as large as we want

shuf -n 100 /home/wordIDs_mapped_to_posting_lists.txt > /home/setalgebra_query_set.txt

Intersection server

./<intersection_server> <IP address:Port Number> <path to dataset> <num of cores: -1 if you want all cores on the machine> <intersection server number> <number of intersection servers in the system>

cd /MicroSuite/src/SetAlgebra/intersection_service/service/
./intersection_server 0.0.0.0:50050 /home/setalgebra_shrad00.txt 1 1 1

Mid tier

<./union_server> <number of intersection servers> <intersection server ips file> <ip:port number> <union parallelism> <union parallelism> <union parallelism> <dispatch parallelism> <number of response threads>

cd /MicroSuite/src/SetAlgebra/union_service/service/
touch lookup_servers_IP.txt
echo "0.0.0.0:50050" > lookup_servers_IP.txt
./mid_tier_server 1 lookup_servers_IP.txt 0.0.0.0:50051 1 1 1

Client

./<loadgen_union_client> <queries file path> <result file path> <Time to run the program> <QPS> <IP to bind to>

cd /MicroSuite/src/SetAlgebra/load_generator
mkdir ./results
./load_generator_open_loop /home/setalgebra_query_set.txt ./results 30 1000 0.0.0.0:50051

** Recommend **

Dataset for Recommend

wget https://www.mlpack.org/datasets/ml-20m/ratings-only.csv.gz
gunzip ratings-only.csv.gz
mv ./ratings-only.csv /home/user_to_movie_ratings.csv

Split dataset to multiple shards, one per cf server

(if only one cf_server then use whole file).

In this example we split in 100 shards (replace shards_num=100 below with number of shards you would like)

rm /home/user_to_movie_ratings_shard*.txt;shards_num=100;split -d --additional-suffix=.txt -l $(($(($(wc -l < /home/user_to_movie_ratings.csv)+shards_num-1))/shards_num)) /home/user_to_movie_ratings.csv /home/user_to_movie_ratings_shard

Library to process the csv input file to create records of user,movie that have no rating

sudo apt-get install -y libtext-csv-perl

Run the script to produce the combinations of users and movies that have no rating

perl missingmovies.pl ratings-only.csv

move the resulted file to home

mv ./missingmovies.csv /home/missingmovies.csv

Produce recommend_query_set.txt of N random lines (100 in this example) from the missingmovies.csv dataset.

Client query set can be as large as we want

sed 1d /home/missingmovies.csv | shuf -n 100 > /home/recommend_query_set.csv

Server for shard 01

./<cf_server> <Mode 1 - read dataset from text file OR Mode 2 - read dataset from binary file > <num of cores: -1 if you want all cores on the machine>

cd /MicroSuite/src/Recommend/cf_service/service
./cf_server /home/user_to_movie_ratings_shard00.txt 0.0.0.0:50050 1 1 0 1

Midtier

<./recommender_server> <ip:port number> <dispatch_parallelism> <number_of_response_threads>

cd /MicroSuite/src/Recommend/recommender_service/service/
touch lookup_servers_IP.txt
echo "0.0.0.0:50050" > lookup_servers_IP.txt
./mid_tier_server 1 lookup_servers_IP.txt 0.0.0.0:50051 1 1 1

Load

./<loadgen_recommender_client>

cd /MicroSuite/src/Recommend/load_generator/
mkdir ./results
./load_generator_open_loop /home/recommend_query_set.csv results 30 1 0.0.0.0:50051

(5) ** Commands used to compile the benchmarks and prepare the docker image **

Install dependencies for µSuite

su
apt-get update
apt-get -y install build-essential autoconf libtool curl cmake git pkg-config
apt-get -y install libz-dev
apt-get -y install nano
apt-get -y install wget
apt-get -y install npm
npm install -g @bazel/bazelisk

grpc

git clone -b v1.26.0 https://github.com/grpc/grpc
cd grpc
git submodule update --init
nano src/core/lib/debug/trace.cc

CHANGE

void TraceFlagList::Add(TraceFlag* flag) {
  flag->next_tracer_ = root_tracer_;
  root_tracer_ = flag;
}

_CHANGE

TO

void TraceFlagList::Add(TraceFlag* flag) {
  for (TraceFlag* t = root_tracer_; t != nullptr; t = t->next_tracer_) {
    if (t == flag) {
      return;
    }
  }
  flag->next_tracer_ = root_tracer_;
  root_tracer_ = flag;

_TO

make
make install
cd ../

protobuf

wget https://github.com/protocolbuffers/protobuf/releases/download/v3.8.0/protobuf-cpp-3.8.0.tar.gz
tar -xzvf protobuf-cpp-3.8.0.tar.gz
cd protobuf-3.8.0/
./configure
make
make check
make install
ldconfig
cd ../

OpenSSL and Intel's MKL

apt-get -y install openssl
apt-get -y install libssl-dev
apt-get -y install cpio
wget https://registrationcenter-download.intel.com/akdlm/irc_nas/tec/12725/l_mkl_2018.2.199.tgz
tar xzvf l_mkl_2018.2.199.tgz
cd l_mkl_2018.2.199
./install.sh
cd ../

FLAN

cd /MicroSuite/src/HDSearch/mid_tier_service/
mkdir build
cd build
cmake ..
make install
make

MLPACK

apt-get -y install libmlpack-dev

HDSearch benchmark

cd /MicroSuite/src/HDSearch/protoc_files
make

cd /MicroSuite/src/HDSearch/bucket_service/service/helper_files

nano server_helper.cc
#include <grpc/grpc.h>
#include <grpcpp/server.h>
#include <grpcpp/server_builder.h>

nano client_helper.cc
#include <grpcpp/channel.h>
#include <grpcpp/client_context.h>
#include <grpc/status.h>

cd ../
make

cd ../../mid_tier_service/service/
apt-get install libboost-all-dev
apt-get install sudo -y
make

cd ../../load_generator/
make

Router benchmark

cd /MicroSuite/src/Router/protoc_files
make clean
make

cd ../lookup_service/service
apt -y install libmemcached-dev
make

cd ../../mid_tier_service/service/
nano ../../lookup_service/service/helper_files/client_helper.cc
#include <grpcpp/channel.h>
make

cd ../../load_generator/
make

Memcached installation for this benchmark

Guide: https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-memcached-on-ubuntu-18-04

apt install memcached
apt install libmemcached-tools
apt install systemd
service memcached restart

Setalgebra benchmark

cd /MicroSuite/src/SetAlgebra/protoc_files
make clean
make

cd ../intersection_service/service/
make

cd ../../union_service/service
nano ../../intersection_service/service/helper_files/client_helper.cc
#include <grpcpp/channel.h>
make

cd ../../load_generator/
make

Recommend benchmark

cd /MicroSuite/src/Recommend/protoc_files
make

cd /MicroSuite/src/Recommend/cf_service/service/
nano +280 cf_server.cc
cf_matrix = new CF(dataset, amf::NMFALSFactorizer(), 5, 5);    
//cf_matrix->Init();

nano helper_files/client_helper.cc
#include <grpcpp/channel.h>
make

cd ../../recommender_service/service/
make

cd ../../load_generator/
make

(6) ** Commands used to compile and run the single node client and midtier**

All benchmarks

To compile the single node client you must:
1) rename the *_singlenode source code under benchmarkname/load_generator/ and benchmarkname/load_generator/helper_files to its original name (remove singlenode from name).
2) make clean
3) make

To compile the single node mid_tier you must:
1) rename the *_singlenode source code under benchmarkname/mid_tier/ to its original name (remove singlenode from name)
2) make clean
3) make

To run single node experiments:
- Nothing changes for mid_tier and bucket services for all benchmarks
- Client accepts an additional parameter (last parameter). The core ID that the client will monitor for C6 entry/exit