Delegation broken?

[Lloyd, Dave]

I'm attempting to implement delegation with func 0.28. So far I've:

. Added Delegate = True to /etc/func/overlord.conf on the master overlord server
. Run func-build-map

Currently all hosts are directly pingable by the overlord. When I run something like func "*" ping I get this:

Traceback (most recent call last):
File "/usr/bin/func", line 28, in ?
ret = cli.parse(argv)
File "/usr/lib/python2.4/site-packages/func/overlord/command.py", line 252, in parse
return self.subCommands[command].parse(args[1:])
File "/usr/lib/python2.4/site-packages/func/overlord/command.py", line 233, in parse
ret = self.do(args)
File "/usr/lib/python2.4/site-packages/func/overlord/cmd_modules/ping.py", line 66, in do
results = overlord_obj.run("test", "ping", [])
File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 791, in run
directhash.update(self.run_direct(module,method,args,nforks))
TypeError: iteration over non-sequence

I added some pretty printing to the args:
module:
'test'
method:
'ping'
args:
[]
nforks:
1

If I do something more complicated, here's what I get:
func "*" call command run "cat /etc/hosts"
module:
'jobs'
method:
'job_status'
args:
('1308912977.796803-minion',)
nforks:
1
Traceback (most recent call last):
File "/usr/bin/func", line 28, in ?
ret = cli.parse(argv)
File "/usr/lib/python2.4/site-packages/func/overlord/command.py", line 252, in parse
return self.subCommands[command].parse(args[1:])
File "/usr/lib/python2.4/site-packages/func/overlord/cmd_modules/call.py", line 227, in parse
return base_command.BaseCommand.parse(self, argv)
File "/usr/lib/python2.4/site-packages/func/overlord/command.py", line 233, in parse
ret = self.do(args)
File "/usr/lib/python2.4/site-packages/func/overlord/cmd_modules/call.py", line 348, in do
return self.overlord_obj.local.utils.async_poll(results, self.print_results)
File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 90, in __call__
return self.clientref.run(module,method,args,nforks=self.nforks)
File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 749, in run
return self.methods[method](*args)
File "/usr/lib/python2.4/site-packages/func/overlord/modules/utils.py", line 15, in async_poll
(return_code, async_results) = self.parent.job_status(job_id)
File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 605, in job_status
client_class_config=self.config)
File "/usr/lib/python2.4/site-packages/func/jobthing.py", line 287, in job_status
minion_result = client.jobs.job_status(minion_job)
File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 90, in __call__
return self.clientref.run(module,method,args,nforks=self.nforks)
File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 791, in run
directhash.update(self.run_direct(module,method,args,nforks))
TypeError: cannot convert dictionary update sequence element #0 to a sequence

If I turn off delegation, all works well.

Anyone have any ideas about what's going on and how I can further debug the issue?

--
Dave Lloyd
Product Support Engineer, Exegy, Inc.
349 Marshall Road, Suite 100
St. Louis, MO 63119 USA
+1.314.218.3600 extension 500
Support: +1.314.218.3555
dlloyd-***@public.gmane.org



This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others.

[Greg Swift]

Post by Lloyd, Dave
. Added Delegate = True to /etc/func/overlord.conf on the master overlord server
. Run func-build-map
 File "/usr/bin/func", line 28, in ?
   ret = cli.parse(argv)
 File "/usr/lib/python2.4/site-packages/func/overlord/command.py", line 252, in parse
   return self.subCommands[command].parse(args[1:])
 File "/usr/lib/python2.4/site-packages/func/overlord/command.py", line 233, in parse
   ret = self.do(args)
 File "/usr/lib/python2.4/site-packages/func/overlord/cmd_modules/ping.py", line 66, in do
   results = overlord_obj.run("test", "ping", [])
 File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 791, in run
   directhash.update(self.run_direct(module,method,args,nforks))
TypeError: iteration over non-sequence
'test'
'ping'
[]
1
func "*" call command run "cat /etc/hosts"
'jobs'
'job_status'
('1308912977.796803-minion',)
1
 File "/usr/bin/func", line 28, in ?
   ret = cli.parse(argv)
 File "/usr/lib/python2.4/site-packages/func/overlord/command.py", line 252, in parse
   return self.subCommands[command].parse(args[1:])
 File "/usr/lib/python2.4/site-packages/func/overlord/cmd_modules/call.py", line 227, in parse
   return base_command.BaseCommand.parse(self, argv)
 File "/usr/lib/python2.4/site-packages/func/overlord/command.py", line 233, in parse
   ret = self.do(args)
 File "/usr/lib/python2.4/site-packages/func/overlord/cmd_modules/call.py", line 348, in do
   return self.overlord_obj.local.utils.async_poll(results, self.print_results)
 File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 90, in __call__
   return self.clientref.run(module,method,args,nforks=self.nforks)
 File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 749, in run
   return self.methods[method](*args)
 File "/usr/lib/python2.4/site-packages/func/overlord/modules/utils.py", line 15, in async_poll
   (return_code, async_results) = self.parent.job_status(job_id)
 File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 605, in job_status
   client_class_config=self.config)
 File "/usr/lib/python2.4/site-packages/func/jobthing.py", line 287, in job_status
   minion_result = client.jobs.job_status(minion_job)
 File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 90, in __call__
   return self.clientref.run(module,method,args,nforks=self.nforks)
 File "/usr/lib/python2.4/site-packages/func/overlord/client.py", line 791, in run
   directhash.update(self.run_direct(module,method,args,nforks))
TypeError: cannot convert dictionary update sequence element #0 to a sequence
If I turn off delegation, all works well.
Anyone have any ideas about what's going on and how I can further debug the issue?

So... in my environment I have 4 levels of overlords. I'll be fancy
with the names to help distinguish layers: (paste this into a
monospace editor if it doesn't line up)

emperor
|
overlord____________
| \
lord1_____ lord2_____
| \ | \
foreman1 servants1 foreman2 servants2
| |
fieldhands1 fieldhands2

The only level I can successfully run delegated calls from almost all
the time (except for map generation, which sometimes takes so long I
have to manually build it) is: emperor

If I run delegation at the lordX level, and my glob matches the
servants and fieldhands, It will error (different than the one you
have). But as long as its grabbing just from servants OR fieldhands
its fine.

I can't remember the exact scenario that describes what you are seeing
(i've run into it as well), but it has to do with errors coming from
hosts saying they can't find the glob you called.

Its kind of irritating, but without the deep lays of delegation I have
none of it seems to work.

This is something that is on my list to track down and resolve, but I
need to finish the implementation of a big project before I can work
on try to finish tracking it down.

[Lloyd, Dave]

-----Original Message-----
Sent: Friday, June 24, 2011 10:44 AM
To: Lloyd, Dave
Subject: Re: [Func-list] Delegation broken?

Post by Lloyd, Dave
. Added Delegate = True to /etc/func/overlord.conf on the master

overlord server

Post by Lloyd, Dave
. Run func-build-map
Currently all hosts are directly pingable by the overlord. When I run
File "/usr/bin/func", line 28, in ?
ret = cli.parse(argv)
File "/usr/lib/python2.4/site-packages/func/overlord/command.py",

line 252, in parse

Post by Lloyd, Dave
return self.subCommands[command].parse(args[1:])
File "/usr/lib/python2.4/site-packages/func/overlord/command.py",

line 233, in parse

Post by Lloyd, Dave
ret = self.do(args)
File "/usr/lib/python2.4/site-

packages/func/overlord/cmd_modules/ping.py", line 66, in do

Post by Lloyd, Dave
results = overlord_obj.run("test", "ping", [])
File "/usr/lib/python2.4/site-packages/func/overlord/client.py",

line 791, in run

Post by Lloyd, Dave
directhash.update(self.run_direct(module,method,args,nforks))
TypeError: iteration over non-sequence
'test'
'ping'
[]
1
func "*" call command run "cat /etc/hosts"
'jobs'
'job_status'
('1308912977.796803-minion',)
1
File "/usr/bin/func", line 28, in ?
ret = cli.parse(argv)
File "/usr/lib/python2.4/site-packages/func/overlord/command.py",

line 252, in parse

Post by Lloyd, Dave
return self.subCommands[command].parse(args[1:])
File "/usr/lib/python2.4/site-

packages/func/overlord/cmd_modules/call.py", line 227, in parse

Post by Lloyd, Dave
return base_command.BaseCommand.parse(self, argv)
File "/usr/lib/python2.4/site-packages/func/overlord/command.py",

line 233, in parse

Post by Lloyd, Dave
ret = self.do(args)
File "/usr/lib/python2.4/site-

packages/func/overlord/cmd_modules/call.py", line 348, in do

Post by Lloyd, Dave
return self.overlord_obj.local.utils.async_poll(results,

self.print_results)

Post by Lloyd, Dave
File "/usr/lib/python2.4/site-packages/func/overlord/client.py",

line 90, in __call__

Post by Lloyd, Dave
return self.clientref.run(module,method,args,nforks=self.nforks)
File "/usr/lib/python2.4/site-packages/func/overlord/client.py",

line 749, in run

Post by Lloyd, Dave
return self.methods[method](*args)
File "/usr/lib/python2.4/site-

packages/func/overlord/modules/utils.py", line 15, in async_poll

Post by Lloyd, Dave
(return_code, async_results) = self.parent.job_status(job_id)
File "/usr/lib/python2.4/site-packages/func/overlord/client.py",

line 605, in job_status

Post by Lloyd, Dave
client_class_config=self.config)
File "/usr/lib/python2.4/site-packages/func/jobthing.py", line 287,

in job_status

Post by Lloyd, Dave
minion_result = client.jobs.job_status(minion_job)
File "/usr/lib/python2.4/site-packages/func/overlord/client.py",

line 90, in __call__

Post by Lloyd, Dave
return self.clientref.run(module,method,args,nforks=self.nforks)
File "/usr/lib/python2.4/site-packages/func/overlord/client.py",

line 791, in run

Post by Lloyd, Dave
directhash.update(self.run_direct(module,method,args,nforks))
TypeError: cannot convert dictionary update sequence element #0 to a

sequence

Post by Lloyd, Dave
If I turn off delegation, all works well.
Anyone have any ideas about what's going on and how I can further

debug the issue?
So... in my environment I have 4 levels of overlords. I'll be fancy
with the names to help distinguish layers: (paste this into a
monospace editor if it doesn't line up)
emperor
|
overlord____________
| \
lord1_____ lord2_____
| \ | \
foreman1 servants1 foreman2 servants2
| |
fieldhands1 fieldhands2
The only level I can successfully run delegated calls from almost all
the time (except for map generation, which sometimes takes so long I
have to manually build it) is: emperor
If I run delegation at the lordX level, and my glob matches the
servants and fieldhands, It will error (different than the one you
have). But as long as its grabbing just from servants OR fieldhands
its fine.
I can't remember the exact scenario that describes what you are seeing
(i've run into it as well), but it has to do with errors coming from
hosts saying they can't find the glob you called.
Its kind of irritating, but without the deep lays of delegation I have
none of it seems to work.
This is something that is on my list to track down and resolve, but I
need to finish the implementation of a big project before I can work
on try to finish tracking it down.

I'm not even able to get to the first level of overlords. If I turn delegation on at all I get the failures mentioned above. Setting delegate=false on the emperor and things work again. Ultimately, my goal is the same as yours except the fieldhands and foremen may not have direct network access back to the emperor.

Are there other configuration settings I need to be implementing on emperor and other systems? Should I set delegate = True on all systems in /etc/func/overlord.conf or do I only need to set it on the emperor systems (I've only set it on the emperor system so far)? Unfortunately the documentation is a bit light on setting up delegation.

--
Dave Lloyd
Product Support Engineer, Exegy, Inc.


This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others.