Discussion:
Certmaster Segfault & Groups issue
Andrew Rankin
2010-09-24 14:37:17 UTC
Permalink
Hi.

So, two more issues:

Seems Certmaster is segfaulting when trying to sign certs:

[***@rtpus175 ~]# certmaster-ca -s node201.175.csc.local
Segmentation fault

[***@rtpus175 ~]# certmaster-ca --version
version: 0.26
release: 1
source build date: Tue Sep 21 13:35:14 EDT 2010
git commit: 31eab6652863c93d1040065453a6c2a25126f1fd
git date: Mon Sep 20 14:39:34 2010 -0400

-----

When using groups, func appears to be stopping after the first machine:

[***@rtpus040 ~]# func @bc1 ping
[ ok ... ] node101.040.csc.local

[***@rtpus040 ~]# func "*" group --la
Group : bc4
Host : node401.040.csc.local
Host : node402.040.csc.local
Host : node403.040.csc.local
Host : node404.040.csc.local
Host : node405.040.csc.local
Host : node406.040.csc.local
Host : node407.040.csc.local
Host : node408.040.csc.local
Host : node409.040.csc.local
Host : node410.040.csc.local
Host : node411.040.csc.local
Host : node412.040.csc.local
Host : node413.040.csc.local
Host : node414.040.csc.local
Group : bc1
Host : node101.040.csc.local
Host : node102.040.csc.local
Host : node103.040.csc.local
Host : node104.040.csc.local
Host : node105.040.csc.local
Host : node106.040.csc.local
Host : node107.040.csc.local
Host : node108.040.csc.local
Host : node109.040.csc.local
Host : node110.040.csc.local
Host : node111.040.csc.local
Host : node112.040.csc.local
Host : node113.040.csc.local
Host : node114.040.csc.local
Group : bc2
Host : node201.040.csc.local
Host : node202.040.csc.local
Host : node203.040.csc.local
Host : node204.040.csc.local
Host : node205.040.csc.local
Host : node206.040.csc.local
Host : node207.040.csc.local
Host : node208.040.csc.local
Host : node209.040.csc.local
Host : node210.040.csc.local
Host : node211.040.csc.local
Host : node212.040.csc.local
Host : node213.040.csc.local
Host : node214.040.csc.local
Group : bc3
Host : node301.040.csc.local
Host : node302.040.csc.local
Host : node303.040.csc.local
Host : node304.040.csc.local
Host : node305.040.csc.local
Host : node306.040.csc.local
Host : node307.040.csc.local
Host : node308.040.csc.local
Host : node309.040.csc.local
Host : node310.040.csc.local
Host : node311.040.csc.local
Host : node312.040.csc.local
Host : node313.040.csc.local
Host : node314.040.csc.local

Thanks,
Andrew
seth vidal
2010-09-24 14:47:13 UTC
Permalink
Post by Andrew Rankin
Hi.
Segmentation fault
That's exciting - can you strace that?
Post by Andrew Rankin
[ ok ... ] node101.040.csc.local
Group : bc4
Host : node401.040.csc.local
Host : node402.040.csc.local
hmm weird spacing going on...

Can you attach your full groups config file?

-sv
Andrew Rankin
2010-09-24 14:56:50 UTC
Permalink
Post by seth vidal
Post by Andrew Rankin
Hi.
Segmentation fault
That's exciting - can you strace that?
Post by Andrew Rankin
[ ok ... ] node101.040.csc.local
Group : bc4
         Host : node401.040.csc.local
         Host :  node402.040.csc.local
hmm weird spacing going on...
Can you attach your full groups config file?
-sv
Hey Seth,

Your spacing comment made me check the groups file, if there is a
space after the semicolon it breaks. Remove the space, all is well.
Since that groups file has been in place for about a year, that is
new. The file format description at
https://fedorahosted.org/func/wiki/GroupsApi has spaces too.

I've attached the strace.

Thanks,
Andrew
seth vidal
2010-09-24 15:35:16 UTC
Permalink
Post by Andrew Rankin
Post by seth vidal
Post by Andrew Rankin
Hi.
Segmentation fault
That's exciting - can you strace that?
Wow, that's bizarre. It seems to be dying right as it has finished
everything else.

what distro/ver/python-ver/openssl version is this?
Post by Andrew Rankin
Your spacing comment made me check the groups file, if there is a
space after the semicolon it breaks. Remove the space, all is well.
Since that groups file has been in place for about a year, that is
new. The file format description at
https://fedorahosted.org/func/wiki/GroupsApi has spaces too.
/me goes to fix

thanks
-sv
seth vidal
2010-09-24 15:54:11 UTC
Permalink
Post by seth vidal
Post by Andrew Rankin
Post by seth vidal
Post by Andrew Rankin
Hi.
Segmentation fault
That's exciting - can you strace that?
Wow, that's bizarre. It seems to be dying right as it has finished
everything else.
what distro/ver/python-ver/openssl version is this?
Post by Andrew Rankin
Your spacing comment made me check the groups file, if there is a
space after the semicolon it breaks. Remove the space, all is well.
Since that groups file has been in place for about a year, that is
new. The file format description at
https://fedorahosted.org/func/wiki/GroupsApi has spaces too.
/me goes to fix
Can you test a really simple patch for me for the groups thing?

-sv
Andrew Rankin
2010-09-24 16:23:23 UTC
Permalink
Post by seth vidal
Wow, that's bizarre. It seems to be dying right as it has finished
everything else.
what distro/ver/python-ver/openssl version is this?
[***@rtpus175 ~]# python
Python 2.4.3 (#1, Sep 3 2009, 15:37:37)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

[***@rtpus175 ~]# cat /etc/redhat-release
CentOS release 5.5 (Final)

[***@rtpus175 ~]# uname -a
Linux rtpus175.corpnet2.com 2.6.18-194.11.4.el5 #1 SMP Tue Sep 21
05:04:09 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

[***@rtpus175 ~]# openssl
OpenSSL> version
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
Post by seth vidal
Post by Andrew Rankin
Your spacing comment made me check the groups file, if there is a
space after the semicolon it breaks.  Remove the space, all is well.
Since that groups file has been in place for about a year, that is
new.  The file format description at
https://fedorahosted.org/func/wiki/GroupsApi has spaces too.
/me goes to fix
thanks
-sv
I'll be happy to test a patch.

Thanks,
Andrew
seth vidal
2010-09-24 16:25:34 UTC
Permalink
Post by Andrew Rankin
Post by seth vidal
Wow, that's bizarre. It seems to be dying right as it has finished
everything else.
what distro/ver/python-ver/openssl version is this?
Python 2.4.3 (#1, Sep 3 2009, 15:37:37)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
CentOS release 5.5 (Final)
Linux rtpus175.corpnet2.com 2.6.18-194.11.4.el5 #1 SMP Tue Sep 21
05:04:09 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
OpenSSL> version
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
Post by seth vidal
Post by Andrew Rankin
Your spacing comment made me check the groups file, if there is a
space after the semicolon it breaks. Remove the space, all is well.
Since that groups file has been in place for about a year, that is
new. The file format description at
https://fedorahosted.org/func/wiki/GroupsApi has spaces too.
/me goes to fix
thanks
-sv
I'll be happy to test a patch.
Patch:

diff --git a/func/overlord/group/conf_backend.py
b/func/overlord/group/conf_backend.py
index 70ad56c..6c2de21 100644
--- a/func/overlord/group/conf_backend.py
+++ b/func/overlord/group/conf_backend.py
@@ -61,6 +61,7 @@ class ConfBackend(BaseBackend):
hosts = hosts.replace(';',',')
hosts = hosts.split(",")
for h in hosts:
+ h = h.strip()
self.add_host_to_group(section,h,save=False)

def add_host_to_group(self,group,host,save=True):


only apply that on the overlord(s)

see if it magically fixes your problems.

-sv
Andrew Rankin
2010-09-24 16:31:24 UTC
Permalink
Post by seth vidal
Post by seth vidal
Wow, that's bizarre. It seems to be dying right as it has finished
everything else.
what distro/ver/python-ver/openssl version is this?
Python 2.4.3 (#1, Sep  3 2009, 15:37:37)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
CentOS release 5.5 (Final)
Linux rtpus175.corpnet2.com 2.6.18-194.11.4.el5 #1 SMP Tue Sep 21
05:04:09 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
OpenSSL> version
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
Post by seth vidal
Post by Andrew Rankin
Your spacing comment made me check the groups file, if there is a
space after the semicolon it breaks.  Remove the space, all is well.
Since that groups file has been in place for about a year, that is
new.  The file format description at
https://fedorahosted.org/func/wiki/GroupsApi has spaces too.
/me goes to fix
thanks
-sv
I'll be happy to test a patch.
diff --git a/func/overlord/group/conf_backend.py
b/func/overlord/group/conf_backend.py
index 70ad56c..6c2de21 100644
--- a/func/overlord/group/conf_backend.py
+++ b/func/overlord/group/conf_backend.py
                    hosts = hosts.replace(';',',')
                    hosts = hosts.split(",")
+                        h = h.strip()
                        self.add_host_to_group(section,h,save=False)
only apply that on the overlord(s)
see if it magically fixes your problems.
-sv
Much better:

[***@rtpus040 group]# func @bc1 ping
[ ok ... ] node101.040.csc.local
[ ok ... ] node111.040.csc.local
[ ok ... ] node108.040.csc.local
[ ok ... ] node109.040.csc.local
[ ok ... ] node114.040.csc.local
[ ok ... ] node106.040.csc.local
[ ok ... ] node113.040.csc.local
[ ok ... ] node107.040.csc.local
[ ok ... ] node110.040.csc.local
[ ok ... ] node112.040.csc.local
[ ok ... ] node102.040.csc.local
[ ok ... ] node103.040.csc.local
[ ok ... ] node104.040.csc.local
[ ok ... ] node105.040.csc.local

Thanks,
Andrew
seth vidal
2010-09-24 16:39:16 UTC
Permalink
Post by Andrew Rankin
Post by seth vidal
Post by Andrew Rankin
Post by seth vidal
Wow, that's bizarre. It seems to be dying right as it has finished
everything else.
what distro/ver/python-ver/openssl version is this?
Python 2.4.3 (#1, Sep 3 2009, 15:37:37)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
CentOS release 5.5 (Final)
Linux rtpus175.corpnet2.com 2.6.18-194.11.4.el5 #1 SMP Tue Sep 21
05:04:09 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
OpenSSL> version
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
Post by seth vidal
Post by Andrew Rankin
Your spacing comment made me check the groups file, if there is a
space after the semicolon it breaks. Remove the space, all is well.
Since that groups file has been in place for about a year, that is
new. The file format description at
https://fedorahosted.org/func/wiki/GroupsApi has spaces too.
/me goes to fix
thanks
-sv
I'll be happy to test a patch.
diff --git a/func/overlord/group/conf_backend.py
b/func/overlord/group/conf_backend.py
index 70ad56c..6c2de21 100644
--- a/func/overlord/group/conf_backend.py
+++ b/func/overlord/group/conf_backend.py
hosts = hosts.replace(';',',')
hosts = hosts.split(",")
+ h = h.strip()
self.add_host_to_group(section,h,save=False)
only apply that on the overlord(s)
see if it magically fixes your problems.
-sv
[ ok ... ] node101.040.csc.local
[ ok ... ] node111.040.csc.local
[ ok ... ] node108.040.csc.local
[ ok ... ] node109.040.csc.local
[ ok ... ] node114.040.csc.local
[ ok ... ] node106.040.csc.local
[ ok ... ] node113.040.csc.local
[ ok ... ] node107.040.csc.local
[ ok ... ] node110.040.csc.local
[ ok ... ] node112.040.csc.local
[ ok ... ] node102.040.csc.local
[ ok ... ] node103.040.csc.local
[ ok ... ] node104.040.csc.local
[ ok ... ] node105.040.csc.local
pushed.

-sv
Andrew Rankin
2010-09-24 18:41:27 UTC
Permalink
Post by seth vidal
Wow, that's bizarre. It seems to be dying right as it has finished
everything else.
what distro/ver/python-ver/openssl version is this?
It's segfaulting at certs.py line 145...

xt = crypto.X509Extension('basicConstraints', False ,'CA:FALSE')

There is an old bug on a Debian list that looks similar:

http://www.mail-archive.com/debian-bugs-dist-***@public.gmane.org/msg162203.html

My version is pyOpenSSL-0.6-1.p24.7.2.2.x86_64

Does not look like a certmaster issue anyways.

Andrew
seth vidal
2010-09-24 20:40:25 UTC
Permalink
Post by Andrew Rankin
Post by seth vidal
Wow, that's bizarre. It seems to be dying right as it has finished
everything else.
what distro/ver/python-ver/openssl version is this?
It's segfaulting at certs.py line 145...
xt = crypto.X509Extension('basicConstraints', False ,'CA:FALSE')
My version is pyOpenSSL-0.6-1.p24.7.2.2.x86_64
Does not look like a certmaster issue anyways.
It sort of is - it's something we added that I thought we had tested on
rhel5.


hmm
-sv
Andrew Rankin
2010-09-24 21:51:37 UTC
Permalink
Post by seth vidal
Post by Andrew Rankin
Post by seth vidal
Wow, that's bizarre. It seems to be dying right as it has finished
everything else.
what distro/ver/python-ver/openssl version is this?
It's segfaulting at certs.py line 145...
xt = crypto.X509Extension('basicConstraints', False ,'CA:FALSE')
My version is pyOpenSSL-0.6-1.p24.7.2.2.x86_64
Does not look like a certmaster issue anyways.
It sort of is - it's something we added that I thought we had tested on
rhel5.
hmm
-sv
Tried it on RHEL5.4, CentOS 5.5 and CentOS 4.8, same outcome. It
does not segfault on OSX.

Tried just:

from OpenSSL import crypto
xt = crypto.X509Extension('basicConstraints',0 ,'CA:FALSE')

Andrew
seth vidal
2010-09-24 21:56:54 UTC
Permalink
Post by Andrew Rankin
Post by seth vidal
Post by Andrew Rankin
Post by seth vidal
Wow, that's bizarre. It seems to be dying right as it has finished
everything else.
what distro/ver/python-ver/openssl version is this?
It's segfaulting at certs.py line 145...
xt = crypto.X509Extension('basicConstraints', False ,'CA:FALSE')
My version is pyOpenSSL-0.6-1.p24.7.2.2.x86_64
Does not look like a certmaster issue anyways.
It sort of is - it's something we added that I thought we had tested on
rhel5.
hmm
-sv
Tried it on RHEL5.4, CentOS 5.5 and CentOS 4.8, same outcome. It
does not segfault on OSX.
from OpenSSL import crypto
xt = crypto.X509Extension('basicConstraints',0 ,'CA:FALSE')
change the CA:FALSE
to CA:False

just to humor me

-sv
Andrew Rankin
2010-09-24 21:59:56 UTC
Permalink
Post by seth vidal
Post by seth vidal
Post by Andrew Rankin
Post by seth vidal
Wow, that's bizarre. It seems to be dying right as it has finished
everything else.
what distro/ver/python-ver/openssl version is this?
It's segfaulting at certs.py line 145...
xt = crypto.X509Extension('basicConstraints', False ,'CA:FALSE')
My version is pyOpenSSL-0.6-1.p24.7.2.2.x86_64
Does not look like a certmaster issue anyways.
It sort of is - it's something we added that I thought we had tested on
rhel5.
hmm
-sv
Tried it on RHEL5.4, CentOS 5.5 and CentOS 4.8, same outcome.   It
does not segfault on OSX.
from OpenSSL import crypto
xt = crypto.X509Extension('basicConstraints',0 ,'CA:FALSE')
change the CA:FALSE
to CA:False
just to humor me
-sv
Traceback (most recent call last):
File "<stdin>", line 1, in ?
OpenSSL.crypto.Error: [('X509 V3 routines', 'X509V3_get_value_bool',
'invalid boolean string')]

Andrew
Louis Coilliot
2010-09-27 18:45:46 UTC
Permalink
Hello,

do you have any news about the segfault problem ?

I'm in trouble with this too.

[***@nlaitm01 ~]# uname -a
Linux nlaitm01 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008
x86_64 x86_64 x86_64 GNU/Linux
[***@nlaitm01 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.3 (Tikanga)
[***@nlaitm01 ~]# echo 'version' | openssl; echo
OpenSSL> OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
OpenSSL>
[***@nlaitm01 ~]# grep segfault /var/log/messages
Sep 27 13:15:57 nlaitm01 kernel: certmaster[14686]: segfault at
0000000000000000 rip 0000000000000000 rsp 00007fff4201e668 error 14

Thanks.

Louis Coilliot
Andrew Rankin
2010-09-27 19:06:44 UTC
Permalink
I opened a Red Hat bug over the weekend for pyOpenSSL:
https://bugzilla.redhat.com/show_bug.cgi?id=637398

Andrew
Post by Louis Coilliot
Hello,
do you have any news about the segfault problem ?
I'm in trouble with this too.
Linux nlaitm01 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008
x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 5.3 (Tikanga)
OpenSSL> OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
OpenSSL>
Sep 27 13:15:57 nlaitm01 kernel: certmaster[14686]: segfault at
0000000000000000 rip 0000000000000000 rsp 00007fff4201e668 error 14
Thanks.
Louis Coilliot
_______________________________________________
Func-list mailing list
https://www.redhat.com/mailman/listinfo/func-list
seth vidal
2010-09-27 19:46:49 UTC
Permalink
Post by Andrew Rankin
https://bugzilla.redhat.com/show_bug.cgi?id=637398
okay - I can step around it by checking for the version of pyOpenSSL but
that's about it, afaict.

I'm open to other solutions that don't involve shipping our own version
of pyOpenSSL. :)

-sv
Andrew Rankin
2010-09-28 00:39:43 UTC
Permalink
Post by seth vidal
Post by Andrew Rankin
https://bugzilla.redhat.com/show_bug.cgi?id=637398
okay - I can step around it by checking for the version of pyOpenSSL but
that's about it, afaict.
I'm open to other solutions that don't involve shipping our own version
of pyOpenSSL. :)
-sv
That's likely the easiest option till it's patched.

Patched this one with
http://sourceforge.net/tracker/index.php?func=detail&aid=1166109&group_id=31249&atid=401760,
appears to fixes the segfault on my CentOS 5.5 system:

http://www.balldawg.net/repo/x86_64/pyOpenSSL-0.6-1.p24.7.2.2.1.x86_64.rpm

Not sure how long it'll take Red Hat to patch it...

Andrew
Louis Coilliot
2010-09-30 13:48:13 UTC
Permalink
This patched rpm is fine for me. Thanks.

Louis
Post by Andrew Rankin
Post by seth vidal
Post by Andrew Rankin
https://bugzilla.redhat.com/show_bug.cgi?id=637398
okay - I can step around it by checking for the version of pyOpenSSL but
that's about it, afaict.
I'm open to other solutions that don't involve shipping our own version
of pyOpenSSL. :)
-sv
That's likely the easiest option till it's patched.
Patched this one with
http://sourceforge.net/tracker/index.php?func=detail&aid=1166109&group_id=31249&atid=401760,
http://www.balldawg.net/repo/x86_64/pyOpenSSL-0.6-1.p24.7.2.2.1.x86_64.rpm
Not sure how long it'll take Red Hat to patch it...
Andrew
_______________________________________________
Func-list mailing list
https://www.redhat.com/mailman/listinfo/func-list
seth vidal
2010-10-05 18:59:47 UTC
Permalink
Post by Andrew Rankin
Post by seth vidal
Post by Andrew Rankin
https://bugzilla.redhat.com/show_bug.cgi?id=637398
okay - I can step around it by checking for the version of pyOpenSSL but
that's about it, afaict.
I'm open to other solutions that don't involve shipping our own version
of pyOpenSSL. :)
-sv
That's likely the easiest option till it's patched.
Patched this one with
http://sourceforge.net/tracker/index.php?func=detail&aid=1166109&group_id=31249&atid=401760,
http://www.balldawg.net/repo/x86_64/pyOpenSSL-0.6-1.p24.7.2.2.1.x86_64.rpm
Not sure how long it'll take Red Hat to patch it...
Any news on this?
-sv
Andrew Rankin
2010-10-05 19:39:00 UTC
Permalink
Let me ask, I've not heard much since last week.

Andrew
Post by seth vidal
Post by Andrew Rankin
Post by seth vidal
Post by Andrew Rankin
https://bugzilla.redhat.com/show_bug.cgi?id=637398
okay - I can step around it by checking for the version of pyOpenSSL but
that's about it, afaict.
I'm open to other solutions that don't involve shipping our own version
of pyOpenSSL. :)
-sv
That's likely the easiest option till it's patched.
Patched this one with
http://sourceforge.net/tracker/index.php?func=detail&aid=1166109&group_id=31249&atid=401760,
http://www.balldawg.net/repo/x86_64/pyOpenSSL-0.6-1.p24.7.2.2.1.x86_64.rpm
Not sure how long it'll take Red Hat to patch it...
Any news on this?
-sv
Andrew Rankin
2011-01-03 14:54:02 UTC
Permalink
As of this morning:

Most recent comment: On 2011-01-03 09:47:06, Siddhesh commented:
"Hi,

This case is currently waiting on engineering for them include this
fix in a future version of RHEL-5. As of now this has been proposed
for RHEL-5.7. I will keep you updated of the progress as and when it
happens. Please feel free to revert if you have any concerns regarding
this bug.

Regards,
Siddhesh"
Post by Andrew Rankin
Let me ask, I've not heard much since last week.
Andrew
Post by seth vidal
Post by Andrew Rankin
Post by seth vidal
Post by Andrew Rankin
https://bugzilla.redhat.com/show_bug.cgi?id=637398
okay - I can step around it by checking for the version of pyOpenSSL but
that's about it, afaict.
I'm open to other solutions that don't involve shipping our own version
of pyOpenSSL. :)
-sv
That's likely the easiest option till it's patched.
Patched this one with
http://sourceforge.net/tracker/index.php?func=detail&aid=1166109&group_id=31249&atid=401760,
http://www.balldawg.net/repo/x86_64/pyOpenSSL-0.6-1.p24.7.2.2.1.x86_64.rpm
Not sure how long it'll take Red Hat to patch it...
Any news on this?
-sv
Loading...