Jenkins Docker Container: Ansible fails to connect via SSH when run via Jenkins, connects fine if not ran via Jenkins. Failed ssh: muxserver_listen"

Question

I'm running a Docker container of Jenkins with Ansible in the same image. Ansible is updating another Docker container on the same Docker network.

Useing an ssh key file.

Tested using root user and user jenkins

Both users tested. I can ssh from the Jenkins/Ansible container to the target container using the CLI. I can run the playbook on the CLI and it works, it's only when I run the playbook through Jenkins pipline it failes.

If I run the same command that Jenkins tries, (copied from the Jenkins console). The only change I made was the tmp ssh key to the real key. The temp key comes from Jenkins credentials. The playbook works fine.

--private-key /var/lib/jenkins/workspace/TestJs/ssh9809173197875714932.key Real Key --private-key /var/lib/jenkins/.ssh/jenkins_key

/usr/bin/ansible-playbook /var/lib/jenkins/workspace/TestJs/deploy/playbooks/deployToTarget.yml -i /var/lib/jenkins/workspace/TestJs/deploy/playbooks/inventory.inv --private-key /var/lib/jenkins/.ssh/jenkins_key -u jenkins

The above call on the Jenkins Docker cli works as expected.

To rule out the Jenkins temp key and validate the Jenkins key credentials, I wrote a simple pipline that connects to the host directly and runs a shell script, that connects fine and work as expected.

Pipeline Snippet

stages {
        stage('Deploy on Test') {
      steps {
        ansiblePlaybook( credentialsId: 'TEST_HOST_SSH_KEY',
                 disableHostKeyChecking: true,
                 installation: 'Ansible',
                 inventory: "${WORKSPACE}/deploy/playbooks/inventory.inv",
                 playbook: "${WORKSPACE}/deploy/playbooks/deployToBSI_SMA.yml",
                 colorized: true)
                
        } 
        }
    }

The console shows this error everytime. I admit I am struggling with this one.

PLAY [Deploy On Test] ***************************************************

TASK [Gathering Facts] *********************************************************
[1;31mfatal: [bsi_sma]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: muxserver_listen: link mux listener /var/lib/jenkins/.ansible/cp/119f480311.4lIaQB6LhOC2svmf => /var/lib/jenkins/.ansible/cp/119f480311: Bad file descriptor", "unreachable": true}[0m

PLAY RECAP *********************************************************************
[0;31mbsi_sma[0m                    : ok=0    changed=0    [1;31munreachable=1   [0m failed=0    skipped=0    rescued=0    ignored=0

Any help is much appreciated; I have spent far too many days on this issue.

UPDATE: Originally, tested both users in the CLI, as in -u jenkins or -u root

Both work while I'm in the CLI as root

If I run su jenkins and try running the same command in the CLI, I get the same error.

So the issue appears to be who the command is run under; Jenkins is running as user Jenkins.

UPDATE fixed it: Jenkins was not picking up the Ansible configuration in the playbook

The CLI was of course

I added this to the Jenkins pipeline and it fixed it

environment {
    ANSIBLE_CONFIG = "${WORKSPACE}/deploy/playbooks/ansible.cfg"
    
}

harrypottar · Accepted Answer · 2024-08-15 17:23:08Z

1

Jenkins running /usr/bin/ansible-playbook, was not picking up the Ansible configuration inside the playbook directory.

The CLI was, of course picking up the configuration.

I added this to the Jenkins pipeline, and it fixed it.

environment {
    ANSIBLE_CONFIG = "${WORKSPACE}/deploy/playbooks/ansible.cfg"
}

I also added this to my Ansible config.

ssh_args= -o ControlMaster=no

Now, it works as expected.

However, I'm unsure why it does not work if a control_path is set.

answered Aug 15, 2024 at 17:23

harrypottar

214 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Max Haase · Accepted Answer · 2024-08-15 15:11:57Z

0

This is often related to the control path used by SSH for connection sharing. It looks like an issue with SSH connection multiplexing when Ansible runs in Jenkins.

To fix it, add this to your Ansible configuration (ansible.cfg):

[ssh_connection]
control_path = %(directory)s/%%h-%%p-%%r
control_persist = no

Disabling control_persist avoids the use of control sockets that can cause "Bad file descriptor" issues inside containers.

Make sure that the Jenkins pipeline uses the correct, persistent key (/var/lib/jenkins/.ssh/jenkins_key) rather than a temporary one.

Also, the Jenkins user should have write access to the Ansible control path directory (/var/lib/jenkins/.ansible/cp/).

That should do it.

answered Aug 15, 2024 at 15:11

Max Haase

2041 gold badge2 silver badges10 bronze badges

2 Comments

harrypottar Over a year ago

Thank you, but that didn't work. I added: [ssh_connection] control_path = %(directory)s/%%h-%%p-%%r control_persist = no Check read-write access to /var/lib/jenkins/.ansible/cp/ drwx------ 2 jenkins jenkins 64 Aug 15 11:41 cp Make sure that the Jenkins pipeline uses the correct, persistent key? I copied and pasted the private SSH key text into the Jenkins credentials, this is what is creating the temp key I still get the same error.

Max Haase Over a year ago

Nice! you got it right, which feels good because I was prepared to stay and help you all the way :-) Voting your answer up!

Collectives™ on Stack Overflow

Jenkins Docker Container: Ansible fails to connect via SSH when run via Jenkins, connects fine if not ran via Jenkins. Failed ssh: muxserver_listen"

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related