Some IT problems are easy to solve if one has the ability to control parameters and environments. When there are constraints, however, things can get really tough.
I found myself in such a constrained scenario. I needed to develop a GitLab CI/CD pipeline to install and run a Ruby application kitchen-terraform (GitHub link) on a GitLab CI/CD executor on which I had a very limited set of OS packages (GNU/Linux, Centos-based, yum) I could install. The Ruby version available was very old. In addition, the standard way of installing Ruby gems (language packages) using https://rubygems.org/gems was unavailable. Also, the repository where Ruby source is available is not accessible.
Add to this that I had a work environment available but it was a highly constrained Microsoft Windows system. I could not install Ruby or any other software on that machine. I was able to send myself brief emails from outside systems, however. And, I could download a Ruby source bundle via browser and Ruby gems were available. This Windows system has a install of Python 3.9 with the standard library along with a limited set of additional Python PyPI packages. However, I do have a Linux system that is isolated over which I have full control.
So, let's break this problem down into parts. First, let's build Ruby from source. Via the Windows system I downloaded the Ruby source package ruby-2.7.6.tar.gz and I could insert this into my GitLab git repo, and I started populating the vendor directory with this. Vendor is a typical directory that has meaning to Ruby and its gem ecosystem. So, building Ruby from source is straight forward. Need to install some OS packages - to be able to build Ruby and a few other things I needed to do. Squelch the output of these commands once I know they work well. This will keep the transcript of the GitLab pipeline from being noisy. This will do it:
yum install 'Development Tools' -y &> /dev/null
yum install openssl-devel openssl -y &> /dev/null
tar xf vendor/ruby-2.7.6.tar.gz
cd ruby-2.7.6
./configure --without-rdoc > /dev/null
make > /dev/null
# GitLab CI/CD pipelines run with root as the user
# so no sudo required on this next command
make install > /dev/null
Building Ruby from source takes a bit of time. My pipeline is going to be running regularly so it would be a nice addition if we don't have to do that every run. I cannot permanently change the GitLab executor, though that would be idea. But, GitLab CI/CD has a cache feature, which allows one to save a portion of the directory structure from the prior run of the pipeline and restore this prior to the next run. So, what if we captured the full Ruby build and set that to be cached. Then just run the 'make install'. Seems to make sense.
However, the Ruby build process using a lot of makefiles is a horrendous mess. It actually compiles C code during 'make install'. Yeah, that is badly broken. Beside not building Ruby every time, it would be ideal to not have to even install the Development Tools via yum, because that takes some unnecessary time. So, begin picking apart the makefiles and figure out what 'make install' is actually doing and what can be done to subset these steps. About 8 hours later, I have the minimal set of commands. However, we need a helper makefile to be added to the suite of other makefiles. Let's call this makefile 'only-install-ext.mak' and we will place it at the root directory. This is the contents to put in that file:
${INSTRUBY} --make="$(MAKE)" $(INSTRUBY_ARGS) --install=ext-comm
${INSTRUBY} --make="$(MAKE)" $(INSTRUBY_ARGS) --install=ext-arch
Here is the minimal script to install a previously built Ruby:
yum install make -y > /dev/null
make do-install-bin
make do-install-lib
make -f GNUmakefile -f only-install-ext.make install_ext_special
make do-install-gem
Ok, one more optimization we can do is to reduce the size of the cache GitLab maintains. This means less time restoring the cache. With some iterations, one can determine which directories of the Ruby build are really not needed by deleting the directories and determine if the above install process still works or the directory is needed. These directories are not needed: basictest benchmark bootstraptest cygwin doc sample spec test win32. So, with the full build process, we remove these directories so that, at the end of the pipeline execution, they will not be present to be added to the cache. In the pipeline configuration, we can add this to capture the cache:
build_job:
...
cache:
key: ruby
paths:
- ruby-2.7.6
So, here is the full shell script (bash) for the building/installing Ruby. We start by checking if the cache has been restored. If it has, we just do the "make install" process.
if [ -d ruby-2.7.6 ]; then
cd ruby-2.7.6
yum install make bind-utils openssl -y > /dev/null
make do-install-bin
make do-install-lib
make -f GNUmakefile -f only-install-ext.mak install_ext_special
make do-install-gem
cd ..
else
yum groupinstall 'Development Tools' -y &> /dev/null
yum install openssl-devel bind-utils openssl -y &> /dev/null
tar xf vendor/ruby-2.7.6.tar.gz
cd ruby-2.7.6
./configure --without-rdoc > /dev/null
make > /dev/null
rm -rf basictest benchmark bootstraptest cygwin doc \
sample spec test win32
cd ..
# some additional work done in this else block ...
...
fi
Ok, Ruby building/installing is taken care of in an optimized fashion. Next, if we want to be able to run kitchen-terraform, we need to have some gems to be installed. In fact, it is quite a large number in a very twisted hierarchy of dependencies. Turns out the easy way to get these gems (found after I started down the hard way path). The easy way is to utilize the Linux system over which I have full control to build an exhaustive list of gems with versions. Do this by first installing the very same ruby-2.7.6 version from source (seems all Linux versions are in the dark ages with Ruby versions the OS packages support). Once that is set up, change to a work directory. Create a file called "Gemfile" with this content:
source "https://rubygems.org"
gem "kitchen-terraform", "~> 6.1"
This Gemfile can be used for the Ruby "gem" command but also the super-powered "bundle" command. So, run the bundle command and capture the output.
bundle install > bundle.log
That output file will contain the names and versions of the full set of required gems for kitchen-terraform. Edit this down to a nice file listing one gem name and version per line. So, this file contents can safely be emailed to the restricted Windows system. Now these gems must be downloaded from rubygems.org. But there are so many that you really don't want to do this via your browser. So, Python to the rescue. This will read a file listing gems and versions.
import os
import sys
import requests
def getgem(name, version):
filename = f'{name}-{version}.gem'
url = f'https://rubygems.org/downloads/{filename}'
r = requests.get(url)
with open(filename, 'wb') as fp:
fp.write(r.content)
Now, each of these gems is actually a .tar.gz file. They can be put in the vendor/cache directory in our repo where "bundle" can find them.
Next, create the Gemfile in our repo with this content:
gem "kitchen-terraform", "6.1.0"
Now, about these gems. Most of them are pure ruby code. Nothing special is required to install these gems besides the .tar.gz files downloaded. However, a small number of these gems require compiling C code. That would mean we need to have the yum group 'Development Tools' be installed. We don't want to have the installed routinely because it slows down the pipeline. So, how about if we build those particular gems at the same time we are building Ruby from source and we have the 'Development Tools' installed. Then, we can zip them up and save the result as a GitLab pipeline artifact, which can be subsequently be downloaded and inserted into our GitLab repo.
To do this, we need to install these compiled gems in a special location because there are lots of files created and we want to separate all those from other gems installed when Ruby was built from source. So, the following shell script code will install the gems in the local directory in the subdirectories of: build_info cache doc extensions gems specifications. The zip will then be unpacked on subsequent runs in the system gem directory.
# install these compiled gems from vendor directory and
# capture the zip
for gem in bcrypt_pbkdf-1.1.0 bson-4.15.0 ed25519-1.3.0 \
ffi-1.15.5 unf_ext-0.0.8.2 json-2.6.2;do
gem install -i . -N -V --local "vendor/cache/${gem}.gem" \
> /dev/null
done
# now save results as zip file
zip -r compiled_gems.zip build_info cache doc extensions \
gems specifications > /dev/null
# built it once, set as job artifact and download it
# and store in vendor/
# this is used below when not built here.
compiled_gems="yes"
You will see that compiled_gems is a flag, which if not set, will trigger installing these gems from the zip file later in this code. We change directory to the system gem install directory for unpacking the zip:
# install pre-built gems so we don't need to install
# developer tools every run
if [ -z "$compiled_gems" ]; then
pushd /usr/local/lib/ruby/gems/2.7.0 > /dev/null
unzip $CI_PROJECT_DIR/vendor/compiled_gems.zip > /dev/null
popd > /dev/null
fi
These installations and the others can be tested by querying with the "gem" command:
# check the compiled gems are available
gem list '^(json|unf_ext|bcrypt_pbkdf|bson|ed25519|ffi)' -d
# and a few of the others
gem list '^(aws-eventstream|azure_graph_rbac)' -d
By the way, you can put this in the GitLab pipeline to capture compiled_gems.zip.
build_job:
...
artifacts:
paths:
- compiled_gems.zip
So, if everything is put together, this is the full setup shell script:
if [ -d ruby-2.7.6 ]; then
cd ruby-2.7.6
yum install make bind-utils openssl -y > /dev/null
make do-install-bin
make do-install-lib
make -f GNUmakefile -f only-install-ext.mak install_ext_special
make do-install-gem
cd ..
else
yum groupinstall 'Development Tools' -y &> /dev/null
yum install openssl-devel bind-utils openssl -y &> /dev/null
tar xf vendor/ruby-2.7.6.tar.gz
cd ruby-2.7.6
./configure --without-rdoc > /dev/null
make > /dev/null
rm -rf basictest benchmark bootstraptest cygwin doc sample \
spec test win32
cd ..
# install these compiled gems from vendor directory and
# capture the zip
for gem in bcrypt_pbkdf-1.1.0 bson-4.15.0 ed25519-1.3.0 \
ffi-1.15.5 unf_ext-0.0.8.2 json-2.6.2;do
gem install -i . -N -V --local "vendor/cache/${gem}.gem" \
> /dev/null
done
# now save results as zip file
zip -r compiled_gems.zip build_info cache doc extensions gems \
specifications > /dev/null
# build it once, set as job artifact and download it and
# store in vendor/ this is used below when not built here.
compiled_gems="yes"
fi
# ruby location, /usr/local/bin, is already on path
echo -n 'Ruby version: '
ruby --version
# install pre-built gems so we don't need to install
# developer tools every run
if [ -z "$compiled_gems" ]; then
pushd /usr/local/lib/ruby/gems/2.7.0 > /dev/null
unzip $CI_PROJECT_DIR/vendor/compiled_gems.zip > /dev/null
popd > /dev/null
fi
# install remaining vendored gems; get gems here: https://rubygems.org/gems
bundle install --local &> bundle.log
log_lines=$(wc -l bundle.log|cut -d' ' -f 1)
normal_log_lines=260
if [ $log_lines -ne $normal_log_lines ]; then
echo "--- bundle.log lines=$log_lines ---"
cat bundle.log
fi
# check the compiled gems are available
# gem list '^(json|unf_ext|bcrypt_pbkdf|bson|ed25519|ffi)' -d
# and a few of the others
# gem list '^(aws-eventstream|azure_graph_rbac)' -d
