SSH Key Gotcha with Test Kitchen and OpenStack

When setting up Kitchen to use OpenStack as the provider instead of Vagrant, I encountered a puzzling authentication issue on creation of the instance. I had my public and private SSH key in ~/.ssh/ and they matched the SSH key stored in OpenStack and referenced in the Kitchen configuration. The creation of instances failed with the following error:

$ kitchen create
-----> Starting Kitchen (v1.2.1)
-----> Creating <default-ubuntu-1404>...
 OpenStack instance <88ef6616-04d3-4d0c-a631-8bb0d91a4c63> created.
....................
(server ready)
 Attaching floating IP from <public> pool
 Attaching floating IP <10.0.1.216>
 Waiting for 10.0.1.216:22...
 Waiting for 10.0.1.216:22...
 Waiting for 10.0.1.216:22...
 (ssh ready)
 Using OpenStack keypair <arnes>
 Using public SSH key <~/.ssh/id_rsa.pub>
 Using private SSH key <~/.ssh/id_rsa>
 Adding OpenStack hint for ohai
net.ssh.transport.server_version[3fd08462809c]
net.ssh.transport.algorithms[3fd0846382bc]
net.ssh.authentication.key_manager[3fd08466a064]
net.ssh.authentication.session[3fd08466a8d4]
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #create action: [Authentication failed for user ubuntu@10.0.1.216]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

After some debugging and research my focus turned to the contents of the SSH key files. When generating my keys I originally used PuTTYGen on Windows and saved them in OpenSSH format in addition to PuTTY format. It was the OpenSSH-files I had copied to ~/.ssh/. The format of the public key file was:

---- BEGIN SSH2 PUBLIC KEY ----
Comment: <Comment>
<SSH-key-string>
---- END SSH2 PUBLIC KEY ----

I am most used to the one-line format for public keys used in the authorized_keys file, so I changed the contents of the key file to match the following format:

ssh-rsa <SSH-key-string> <Comment>

Luckily, that was enough for Test Kitchen to work as intended:

$ kitchen create
-----> Starting Kitchen (v1.2.1)
-----> Creating <default-ubuntu-1404>...
 OpenStack instance <c08688f6-a754-4f43-a365-898a38fc06f8> created.
.........................
(server ready)
 Attaching floating IP from <public> pool
 Attaching floating IP <10.0.1.243>
 Waiting for 10.0.1.243:22...
 Waiting for 10.0.1.243:22...
 Waiting for 10.0.1.243:22...
 (ssh ready)
 Using OpenStack keypair <arnes>
 Using public SSH key <~/.ssh/id_rsa.pub>
 Using private SSH key <~/.ssh/id_rsa>
 Adding OpenStack hint for ohai
net.ssh.transport.server_version[3fe8926c1320]
net.ssh.transport.algorithms[3fe8926c06b4]
net.ssh.connection.session[3fe89270b420]
net.ssh.connection.channel[3fe89270b2cc]
Finished creating <default-ubuntu-1404> (0m50.68s).
-----> Kitchen is finished. (0m52.22s)

5 thoughts on “SSH Key Gotcha with Test Kitchen and OpenStack

    1. After installing the driver according to the instructions in the README, I added the necessary config to the global Kitchen config file ~/.kitchen/config.yml. The contents of my config file are:

      ---
      driver:
        name: openstack
        openstack_username: <%= ENV['OS_USERNAME'] %>
        openstack_api_key: <%= ENV['OS_PASSWORD'] %>
        openstack_auth_url: <%= "#{ENV['OS_AUTH_URL']}/tokens" %>
        openstack_tenant: <%= ENV['OS_TENANT_NAME'] %>
        require_chef_omnibus: true
        image_ref: CentOS 7 GC 2014-09-16
        username: centos
        flavor_ref: m1.medium
        key_name: <%= ENV['OS_USERNAME'] %>
        floating_ip_pool: public
        network_ref:
          - NETWORK_NAME
        no_ssh_tcp_check: true
        no_ssh_tcp_check_sleep: 30

      Replace NETWORK_NAME with the actual name of the virtual network you want Kitchen to create VMs on. Also adjust the image_ref, image username (here: centos), flavor_ref, floating_ip_pool and key_name to the correct settings for your OpenStack cluster.

      The references to environment variables are for the variables you get when you source the OpenStack RC file downloaded from Horizon under Access & Security -> API Access. One advantage of this setup is that it’s easy to change OpenStack project used by Kitchen by sourcing the new project’s RC file.

      With this config in place in the global config file, all you need to do in .kitchen.yml in each Chef cookbook repo is to change the default vagrant driver to openstack:

      driver:
        name: openstack

      Like

      1. Ajay K says:

        I deployed openstack using devstack all in one vm

        [root@cstack ~]# kitchen list
        Instance Driver Provisioner Last Action
        default-cirros Openstack ChefSolo
        [root@cstack ~]# kitchen create
        —–> Starting Kitchen (v1.3.1)
        —–> Creating …
        >>>>>> Create failed on instance .
        >>>>>> Please see .kitchen/logs/default-cirros.log for more details
        >>>>>> ——Exception——-
        >>>>>> Class: Kitchen::ActionFailed
        >>>>>> Message: Expected([200, 204]) Actual(400 Bad Request)
        excon.error.response
        :body => “{\”error\”: {\”message\”: \”get_version_v2() got an unexpected keyword argument ‘auth’\”, \”code\”: 400, \”title\”: \”Bad Request\”}}”
        :headers => {
        “Connection” => “close”
        “Content-Length” => “121”
        “Content-Type” => “application/json”
        “Date” => “Fri, 06 Feb 2015 13:49:53 GMT”
        “Server” => “Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5”
        “Vary” => “X-Auth-Token”
        }
        :local_address => “::1”
        :local_port => 49762
        :remote_ip => “::1”
        :status => 400

        >>>>>> ———————-

        content of yml file,
        [root@cstack ~]# cat .kitchen.yml

        driver:
        name: openstack
        openstack_username: admin
        openstack_api_key: password
        openstack_auth_url: http://localhost:35357/v2.0/
        require_chef_omnibus: true
        image_ref: cirros
        flavor_ref: m1.tiny
        openstack_tenant: admin
        server_name: Tcir003001
        network_ref: public
        public_key_path: /opt/stack/.ssh/id_rsa.pub
        private_key_path: /opt/stack/.ssh/id_rsa
        username: cirros

        provisioner:
        name: chef_solo

        platforms:
        # – name: ubuntu-12.04
        # – name: centos-6.4
        – name: cirros

        suites:
        – name: default
        run_list:
        attributes:

        ———————————————————–

        I guess I did not completed certain configuration for kitchen openstack. I dont know what. any clue?

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s