I spent the last few days learning about the openSUSECeph installation process. I ran into some issues, and I’m not
done yet, so these are just my working notes for now. Once complete, I’ll
write up the process on my regular blog.
Prerequisite: build a tool to build and destroy small clusters quickly
I needed a way to quickly provision and destroy
virtual machines that were well suited to run small Ceph clusters. I mostly
run libvirt / kvm
in my home lab, and I didn’t find any solutions tailored to that platform, so
I wrote ceph-libvirt-clusterer.
Ceph-libvirt-clusterer
lets me clone a template virtual machine and attach as many OSD disks
as I’d like in the process. I’m really happy with the tool
considering that I only have a day’s worth of work in it, and I got to
learn some details of the libvirt API and python bindings in the process.
Build a template machine
I built a template machine with
openSUSE’s tumbleweed and
completed the following preliminary configurations:
created ceph user
ceph user has a SSH key
ceph user’s public key is in the ceph user’s authorized_keys file
ceph user is configured for passwordless sudo
emacs is installed (not strictly necessary :-) )
Provision a cluster
I used ceph-libvirt-clusterer to create a four node cluster, and each node had
two 8GBOSD drives attached.
Do you want to reject the key, trust temporarily, or trust always? [r/t/a/? shows all options](r): a Retrieving repository 'ceph' metadata .........................................................................................................................................................................[done] Building repository 'ceph' cache ..............................................................................................................................................................................[done] Loading repository data... Reading installed packages...
S | Name | Summary | Type --+--------------------+---------------------------------------------------+----------- | ceph | User space components of the Ceph file system | package | ceph | User space components of the Ceph file system | srcpackage | ceph-common | Ceph Common | package | ceph-deploy | Admin and deploy tool for Ceph | package | ceph-deploy | Admin and deploy tool for Ceph | srcpackage | ceph-devel-compat | Compatibility package for Ceph headers | package | ceph-fuse | Ceph fuse-based client | package | ceph-libs-compat | Meta package to include ceph libraries | package | ceph-radosgw | Rados REST gateway | package | ceph-test | Ceph benchmarks and test tools | package | libcephfs1 | Ceph distributed file system client library | package | libcephfs1-devel | Ceph distributed file system headers | package | python-ceph-compat | Compatibility package for Cephs python libraries | package | python-cephfs | Python libraries for Ceph distributed file system | package
Bash
First issue: python was missing on the other nodes
When I installed ceph-deploy on the admin node, python was also
installed. The other nodes were still running with a bare minimum
configuration from the tumbleweed install, so python was missing, and
ceph-deploy’s install step failed.
I installed Ansible to correct the problem on all
nodes simultaneously, but Ansible requires python on the remote side, too.
That meant I had to manually install python on the remaining three nodes just
like sysadmins had to do years ago.
Second issue: all nodes need the OBS repository
I didn’t add the OBS repository to the remaining three nodes because I
wanted to see if ceph-deploy would add it automatically. I didn’t expect
that to be the case, but since this version of ceph-deploy came directly from
SUSE, there was a chance.
Fortunately Ansible works now:
ceph@linux-7d21:~/tinyceph> ansible -i ansible-inventory all -a "sudo zypper ar -f http://download.opensuse.org/repositories/filesystems:/ceph/openSUSE_Tumbleweed/ ceph" 192.168.122.122 | success | rc=0 >> Adding repository 'ceph'[......done] Repository 'ceph' successfully added Enabled : Yes Autorefresh : Yes GPG Check : Yes URI : http://download.opensuse.org/repositories/filesystems:/ceph/openSUSE_Tumbleweed/
# and three more nodes worth of output...
ceph@linux-7d21:~/tinyceph> ansible -i ansible-inventory all -a "sudo zypper --gpg-auto-import-keys update"
Bash
Once both of these commands completed, ceph-deploy install worked as expected.
Third issue: I was using IP addresses
ceph-deploy new complains when provided with IP addresses:
ceph@linux-7d21:~/tinyceph> ceph-deploy new 192.168.122.121 192.168.122.122 192.168.122.123 192.168.122.124 usage: ceph-deploy new [-h][--no-ssh-copykey][--fsid FSID] [--cluster-network CLUSTER_NETWORK] [--public-network PUBLIC_NETWORK] MON[MON ...] ceph-deploy new: error: 192.168.122.121 must be a hostname not an IP
Bash
In the future, it’d be pretty cool if ceph-libvirt-clusterer supported
updating DNS records so I didn’t need to resort to the host file
ansible playbook that I used today:
Fourth issue: tumbleweed uses systemd, but ceph-deploy doesn’t expect that
[ceph_deploy.mon][INFO] distro info: openSUSE 20150714 x86_64 [tinyceph-03][DEBUG] determining if provided host has same hostname in remote [tinyceph-03][DEBUG] get remote short hostname [tinyceph-03][DEBUG] deploying mon to tinyceph-03 [tinyceph-03][DEBUG] get remote short hostname [tinyceph-03][DEBUG] remote hostname: tinyceph-03 [tinyceph-03][DEBUG] write cluster configuration to /etc/ceph/{cluster}.conf [tinyceph-03][DEBUG] create the mon path if it does not exist [tinyceph-03][DEBUG] checking for done path: /var/lib/ceph/mon/ceph-tinyceph-03/done [tinyceph-03][DEBUG] create a done file to avoid re-doing the mon deployment [tinyceph-03][DEBUG] create the init path if it does not exist [tinyceph-03][INFO] Running command: sudo /etc/init.d/ceph -c /etc/ceph/ceph.conf start mon.tinyceph-03 [tinyceph-03][ERROR] Traceback (most recent call last): [tinyceph-03][ERROR] File "/usr/lib/python2.7/site-packages/remoto/process.py", line 94, in run [tinyceph-03][ERROR] reporting(conn, result, timeout) [tinyceph-03][ERROR] File "/usr/lib/python2.7/site-packages/remoto/log.py", line 13, in reporting [tinyceph-03][ERROR]received= result.receive(timeout) [tinyceph-03][ERROR] File "/usr/lib/python2.7/site-packages/execnet/gateway_base.py", line 701, in receive [tinyceph-03][ERROR] raise self._getremoteerror() or EOFError() [tinyceph-03][ERROR] RemoteError: Traceback (most recent call last): [tinyceph-03][ERROR] File "<string>", line 1033, in executetask [tinyceph-03][ERROR] File "<remote exec>", line 12, in _remote_run [tinyceph-03][ERROR] File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__ [tinyceph-03][ERROR] errread, errwrite) [tinyceph-03][ERROR] File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child [tinyceph-03][ERROR] raise child_exception [tinyceph-03][ERROR] OSError: [Errno 2] No such file or directory [tinyceph-03][ERROR] [tinyceph-03][ERROR] [ceph_deploy.mon][ERROR] Failed to execute command: /etc/init.d/ceph -c /etc/ceph/ceph.conf start mon.tinyceph-03 [ceph_deploy][ERROR] GenericError: Failed to create 4 monitors
Bash
Sure enough, a little manual inspection revealed no file at /etc/init.d/ceph and systemd integration:
ceph@tinyceph-00:~/tinyceph> ls -la /etc/init.d/ceph ls: cannot access /etc/init.d/ceph: No such file or directory ceph@tinyceph-00:~/tinyceph> sudo service ceph status * ceph.target - ceph target allowing to start/stop all ceph*@.service instances at once Loaded: loaded (/usr/lib/systemd/system/ceph.target; disabled; vendor preset: disabled) Active: inactive (dead)
Jul 19 23:50:35 tinyceph-00 systemd[1]: Reached target ceph target allowing to start/stop all ceph*@.service instances at once. Jul 19 23:50:35 tinyceph-00 systemd[1]: Starting ceph target allowing to start/stop all ceph*@.service instances at once. Jul 19 23:50:47 tinyceph-00 systemd[1]: Stopped target ceph target allowing to start/stop all ceph*@.service instances at once. Jul 19 23:50:47 tinyceph-00 systemd[1]: Stopping ceph target allowing to start/stop all ceph*@.service instances at once. ceph@tinyceph-00:~/tinyceph> sudo service ceph start ceph@tinyceph-00:~/tinyceph> sudo service ceph status * ceph.target - ceph target allowing to start/stop all ceph*@.service instances at once Loaded: loaded (/usr/lib/systemd/system/ceph.target; disabled; vendor preset: disabled) Active: active since Mon 2015-07-20 00:24:01 EDT; 4s ago
Jul 20 00:24:01 tinyceph-00 systemd[1]: Reached target ceph target allowing to start/stop all ceph*@.service instances at once. Jul 20 00:24:01 tinyceph-00 systemd[1]: Starting ceph target allowing to start/stop all ceph*@.service instances at once.
Bash
I learned that this is a known bug,
and I’ll try all of this again with an older version of openSUSE.
… and that’s where I’m calling it a night. I’ll be back at it this week.
Last time I worked on Kallithea’s CI, I got some errors. On a fresh
Ubuntu 14.04 VM without docker, I get the following test results:
In a Virtual Machine
sqlite: 0 errors, 2 skipped
mysql: 0 errors, 2 skipped
postgresql: 1 error, 2 skipped
details:
======================================================================
ERROR: test_index_with_anonymous_access_disabled (kallithea.tests.functional.test_home.TestHomeController)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/packer/src/kallithea-pg/kallithea/tests/functional/test_home.py", line 43, in test_index_with_anonymous_access_disabled
status=302)
File "/home/packer/src/kallithea/.venv/local/lib/python2.7/site-packages/WebTest-1.4.3-py2.7.egg/webtest/app.py", line 759, in get
expect_errors=expect_errors)
File "/home/packer/src/kallithea/.venv/local/lib/python2.7/site-packages/WebTest-1.4.3-py2.7.egg/webtest/app.py", line 1121, in do_request
self._check_status(status, res)
File "/home/packer/src/kallithea/.venv/local/lib/python2.7/site-packages/WebTest-1.4.3-py2.7.egg/webtest/app.py", line 1160, in _check_status
"Bad response: %s (not %s)", res_status, status)
AppError: Bad response: 200 OK (not 302)
----------------------------------------------------------------------
Ran 1482 tests in 311.450s
FAILED (SKIP=2, errors=1)
In a Docker Container
sqlite
I’m betting that these messages are a canary that will help figure out the sqlite failures:
kallithea_1 | not trusting file /code/.hg/hgrc from untrusted user 1000, group 1000
kallithea_1 | not trusting file /tmp/rc_test_lPm4Rl/vcs_test_hg/.hg/hgrc from untrusted user 502, group root
kallithea_1 | not trusting file /tmp/rc_test_lPm4Rl/vcs_test_hg/.hg/hgrc from untrusted user 502, group root
kallithea_1 | not trusting file /tmp/rc_test_lPm4Rl/vcs_test_hg/.hg/hgrc from untrusted user 502, group root
RUN useradd -d /home/kallithea -m -s /bin/bash -u 2000 kallithea
RUN chown -R kallithea /code
USER kallithea
… to ensure that there was no weirdness induced from running as root
and running against files that were owned by a different UID than the
test process, and I got the same four errors. Something’s up when
running in this container.
All the failing tests include the string “non_ascii” in their names.
Let’s see what locale tells us on the virtual machine:
The unit tests now run inside of containers managed by fig. I wrote
two scripts to facilitate the execution:
integration-configs/fig-config-glue.py: reads environment variables set by fig to create a sqlalchemy URL and update an ini file with it.
integration-configs/execute_tests.sh: runs the above script, updating test.ini, then sleeps for 10 seconds while the database starts, then runs nosetests
I had been switching between the various databases just by using a
different fig configuration file, but that is insufficient. Fig must
be invoked with both of the following arguments:
fig -f fig-${DB_TYPE} -p kallithea-${DB_TYPE}
If a project name isn’t specified, fig won’t differentiate between the
various database containers (all named “db” in the configs).
Of course, different numbers of tests fail in each configuration
(including some sqlite tests that don’t fail when I run it directly on
my machine without docker in the middle), so there’s still some
testing and adjusting to complete.
Another annoyance: when the tests complete, fig shuts down both containers,
but fig’s exit code is always zero even if one of the containers exited
with a non-zero return code. Going to ask the team if that’s by design
or if they’re open to changing it. As is, I’ll need to either parse
the nose output or parse the output of fig ps to give an appropriate
exit code to the build server.
I thought I could use ENTRYPOINT to load up environment variables and
do some voodoo with the ini file to get the database connection
configured correctly:
So then anything I ran would run after the /code/.figrc loaded.
Except that doesn’t happen. Read the CMD docs again, and you’ll see
that if the first argument is an executable, it will be executed
via the following: /bin/sh -c, so no /code/.figrc.
/me sighs
/me comes back after 15 minutes
These are not general purpose images. They’re specifically to run test
suites. Why do I care about them working for every case? I can just
write a wrapper script and call it a day.
I spent about an hour tonight working on flexible configurations for
testing Kallithea against various
databases using Fig and
Docker
Fig handles some of the dirty work of linking together Docker
containers. Linked containers get environment variables set to define
endpoints of the other containers. They use
Django as an example
and things look pretty easy: since the configuration file is written
in python, we can just call os.environ.get('DB_1_PORT_5432_TCP_PORT').
No such luck with Pylons and Pyramid, though: there we use an ini file
for configuration. I ran into a few bumps, though.
The paster serve command provides an avenue for command line
configuration: var=value can be repeated to pass in configuration
options on the command line, and the named vars can be referenced in
the ini file with %(var)s. That’s good.
Fig doesn’t seem to support environment variables in it’s YAML
configuration files, so paster serve test.ini
dbhost=$DB_1_PORT_5432_TCP_ADDR results in a literal string
“DB_1_PORT_5432_TCP_ADDR” in the configuration. That’s bad,
but it can be fixed with a wrapper script.
Kallithea’s setup-db command doesn’t support the same var=value
setting on the command line that paster serve supports. That’s bad,
but the wrapper script can rewrite the configuration files rather than
pass in values via arguments. That’s where I’m leaving off for tonight.
One other dangling question: I tried putting my Dockerfile and
Fig yaml configurations in a subdirectory to keep the project root
uncluttered, but it didn’t look like Docker liked using .. in place
of .. I need to confirm that: there’s a chance that something else
was out of line that I didn’t notice.
Each base image type,
defined in Nodepool’s YAML configuration file,
includes a setup attribute that matches one of the scripts. The
matching script is executed in an environment that includes any
NODEPOOL_ variables present in the Nodepol daemon’s environment. From
all that I’ve seen, this typically only includes NODEPOOL_SSH_KEY.
(See jenkins_dev.pp)
So to build replica nodes for personal use, I should just need to copy
the scripts to /opt/nodepool-scripts and execute the right one in my
packer provisioning configuration.
Building a system at work to create disposable machines for developer
and QA use that will match production. The process looks like this:
Query Foreman for Puppet classes used by a hostgroup
Create a Puppetfile with some knowledge about where we keep modules and the names that Foreman provides
Run r10k to sync the modules
I did a bunch of work to extract some metadata from the Modulefiles,
but that was completely unnecessary and thus stupid. Never hurts to
take a step back and draw a process before jumping down a rabbit hole.
I have enough information that I could skip creating the Puppetfile
all together and just clone the repo myself, but I think I like the idea
of using r10k to install modules from the Puppetfile for two reasons:
It ensures that stale modules are purged gracefully. (I would have just scorched the Earth and deleted the entire modules directory)
It produces an artifact that can be used by other standard tools if necessary.
Investigating the
Devstack Gate
documentation. Seems to be more out of date than I originally
expected, or I don’t understand where things run. It mentions a
matrix job called devstack-update-vm-image, but reviewing the
job list shows that
jobs matching that description haven’t run in 9-10 months:
devstack-update-vm-image-hpcloud-az1
devstack-update-vm-image-hpcloud-az2
devstack-update-vm-image-hpcloud-az3
devstack-update-vm-image-rackspace
devstack-update-vm-image-rackspace-dfw
devstack-update-vm-image-rackspace-ord
Looking further, all devstack-update-vm-image and
devstack-check-vms- jobs are disabled.
I suspect that all of this work migrated to the
Nodepool project.
nodepool.openstack.org contains a
directory listing with logs that have recent time stamps. Promising.
The log output
(warning, 20MB) seems to confirm my suspicion.
