Func connections failing

Discussion:

Thomas S Hatch

2011-01-11 00:00:22 UTC

I have been using func for years to manage my systems, but today, while
standing up a new environment I ran into this problem. The func daemons and
the certmaster all start up, and all the certs are made and distributed.
But every time I try to connect via func from the master to a minion th
eminion throws this error:

----------------------------------------
Exception happened during processing of request from ('10.23.0.3', 42188)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 558, in process_request
self.shutdown_request(request)
File "/usr/lib/python2.7/SocketServer.py", line 459, in shutdown_request
request.shutdown(socket.SHUT_WR)
File "/usr/lib/python2.7/site-packages/certmaster/SSLConnection.py", line
54, in shutdown
self.__dict__["conn"].shutdown()
Error: []
----------------------------------------

And the connection fails

I am running the latest func, func --version returns 0.26

Does anyone have any ideas what might be causing this? Or suggestions on how
to find out more information?

The log files are devoid of errors as well.

-Tom Hatch

seth vidal

2011-02-25 21:00:26 UTC

Permalink

Post by Thomas S Hatch
I have been using func for years to manage my systems, but today,
while standing up a new environment I ran into this problem. The func
daemons and the certmaster all start up, and all the certs are made
and distributed.
But every time I try to connect via func from the master to a minion
----------------------------------------
Exception happened during processing of request from ('10.23.0.3', 42188)
File "/usr/lib/python2.7/SocketServer.py", line 558, in
process_request
self.shutdown_request(request)
File "/usr/lib/python2.7/SocketServer.py", line 459, in
shutdown_request
request.shutdown(socket.SHUT_WR)
File "/usr/lib/python2.7/site-packages/certmaster/SSLConnection.py",
line 54, in shutdown
self.__dict__["conn"].shutdown()
Error: []
----------------------------------------
And the connection fails
I am running the latest func, func --version returns 0.26
Does anyone have any ideas what might be causing this? Or suggestions
on how to find out more information?

this is python 2.7, right? Some changes just got committed which help
python 2.7 issues. Can you try those?

I'm sorry it took so long to reply.

-sv

Thomas S Hatch

2011-02-25 21:05:08 UTC

Permalink

Post by seth vidal

this is python 2.7, right? Some changes just got committed which help
python 2.7 issues. Can you try those?
I'm sorry it took so long to reply.
-sv

Hey seth! finally they got committed eh? I have patched Arch Linux func
package to include a patch for this, I am assuming it is the same issue?

https://aur.archlinux.org/packages/func/func/python_27.patch

And yes, I will be happy to test it out, I will roll an Arch package from it
for testing, I will probably get to it tomorrow.

seth vidal

2011-02-25 21:11:06 UTC

Permalink

On Fri, Feb 25, 2011 at 2:00 PM, seth vidal

Post by Thomas S Hatch
I have been using func for years to manage my systems, but

today,

Post by Thomas S Hatch
while standing up a new environment I ran into this problem.

The func

Post by Thomas S Hatch
daemons and the certmaster all start up, and all the certs

are made

Post by Thomas S Hatch
and distributed.
But every time I try to connect via func from the master to

a minion

Post by Thomas S Hatch
----------------------------------------
Exception happened during processing of request from

('10.23.0.3',

Post by Thomas S Hatch
42188)
File "/usr/lib/python2.7/SocketServer.py", line 558, in
process_request
self.shutdown_request(request)
File "/usr/lib/python2.7/SocketServer.py", line 459, in
shutdown_request
request.shutdown(socket.SHUT_WR)
File

"/usr/lib/python2.7/site-packages/certmaster/SSLConnection.py",

Post by Thomas S Hatch
line 54, in shutdown
self.__dict__["conn"].shutdown()
Error: []
----------------------------------------
And the connection fails
I am running the latest func, func --version returns 0.26
Does anyone have any ideas what might be causing this? Or

suggestions

Post by Thomas S Hatch
on how to find out more information?

this is python 2.7, right? Some changes just got committed which help
python 2.7 issues. Can you try those?
I'm sorry it took so long to reply.
-sv
Hey seth! finally they got committed eh? I have patched Arch Linux
func package to include a patch for this, I am assuming it is the same
issue?
https://aur.archlinux.org/packages/func/func/python_27.patch
And yes, I will be happy to test it out, I will roll an Arch package
from it for testing, I will probably get to it tomorrow.

Thomas,
Yes it finally happened. I got the round tuits and got something
together to do a proper test to make sure nothing else had broken.

I'm sorry for the long wait - things got busy but they MIGHT get
better. :)

Your patch is much the same but there was one other one needed to
certmaster/SSLConnection.py which Felix sent in. That makes it work,
too.

-sv

Thomas S Hatch

2011-02-25 21:17:11 UTC

Permalink

Post by seth vidal

On Fri, Feb 25, 2011 at 2:00 PM, seth vidal

Post by Thomas S Hatch
I have been using func for years to manage my systems, but

today,

Post by Thomas S Hatch
while standing up a new environment I ran into this problem.

The func

Post by Thomas S Hatch
daemons and the certmaster all start up, and all the certs

are made

Post by Thomas S Hatch
and distributed.
But every time I try to connect via func from the master to

a minion

Post by Thomas S Hatch
----------------------------------------
Exception happened during processing of request from

('10.23.0.3',

"/usr/lib/python2.7/site-packages/certmaster/SSLConnection.py",

suggestions

Post by Thomas S Hatch
on how to find out more information?

Thomas,
Yes it finally happened. I got the round tuits and got something
together to do a proper test to make sure nothing else had broken.
I'm sorry for the long wait - things got busy but they MIGHT get
better. :)
Your patch is much the same but there was one other one needed to
certmaster/SSLConnection.py which Felix sent in. That makes it work,
too.
-sv

This probably:
https://aur.archlinux.org/packages/certmaster/certmaster/python_27.patch

I sent both patches to the list over a month ago, also, for what it is worth
I have been running func in an Arch Linux environment with over 100 minions
with these patches without issue for the past month.

I hope you get more time to refine func, I think it is great stuff, I know
that ovirt is moving to matahari, and I have had issues with func scaling to
handle the number of minions I have right now in one of my bigger
deployments. But I don't like the matahari architecture as much. I am
working on a func-like project right now called salt that uses zeromq, but I
have not had the cycles to give it the love that it needs.

good luck on func, I am looking forward to the next release, and I will
start testing soon.

seth vidal

2011-02-25 21:28:05 UTC

Permalink

Post by Thomas S Hatch
I hope you get more time to refine func, I think it is great stuff, I
know that ovirt is moving to matahari, and I have had issues with func
scaling to handle the number of minions I have right now in one of my
bigger deployments. But I don't like the matahari architecture as
much. I am working on a func-like project right now called salt that
uses zeromq, but I have not had the cycles to give it the love that it
needs.

what scale problems have you seen? I'm running func on about 145 boxes
and it does pretty well if you bump the number of forks for your
processes.

As far as the amqp-based mechanisms. I think they're great. My biggest
concern is as amqp is used by lots of non-administrative apps (like web
apps, etc) that you'll start running into the problem of your
administrative apps wanting a specific instance/version/kind that will
conflict with what your developers or user-facing apps want.

I'm working on making func easier to setup with even less fuss for
specific tasks.

-sv