When I run the test on my system, it works fine. The nodes are running on debian stable.

work@voyager:~/random/funkload$ fl-run-bench test_example.com.py example.com.test_example.com --distribute --distribute-workers=xxx@node103 -c 1:2:3 -D 30
========================================================================
Benching example.com.test_example.com
========================================================================
test all get operations
------------------------------------------------------------------------

Configuration
=============

* Current time: 2011-10-11T20:34:57.390507
* Configuration file: /home/work/random/funkload/example.com.conf
* Distributed output: log-distributed
* Server: http://flocked.in
* Cycles: [1, 2, 3]
* Cycle duration: 30s
* Sleeptime between request: from 0.0s to 2.0s
* Sleeptime between test case: 1.0s
* Startup delay between thread: 0.2s
* Workers :node103

* Preparing sandboxes for 1 workers..
* Starting 1 workers* Start monitoring 192.168.36.238: ... done.
.

* [node103] returned

* Stop monitoring 192.168.36.238:  done.
Traceback (most recent call last):
  File "/usr/local/bin/fl-run-bench", line 9, in <module>
    load_entry_point('funkload==1.16.1', 'console_scripts', 'fl-run-bench')()
  File "/usr/local/lib/python2.7/dist-packages/funkload-1.16.1-py2.7.egg/funkload/BenchRunner.py", line 707, in main
    ret = distmgr.run()
  File "/usr/local/lib/python2.7/dist-packages/funkload-1.16.1-py2.7.egg/funkload/Distributed.py", line 468, in run
    self.correlate_statistics()
  File "/usr/local/lib/python2.7/dist-packages/funkload-1.16.1-py2.7.egg/funkload/Distributed.py", line 605, in correlate_statistics
    ratio = self._calculate_time_skew(results, stats)
  File "/usr/local/lib/python2.7/dist-packages/funkload-1.16.1-py2.7.egg/funkload/Distributed.py", line 565, in _calculate_time_skew
    monitor_min = min_time(stats)
  File "/usr/local/lib/python2.7/dist-packages/funkload-1.16.1-py2.7.egg/funkload/Distributed.py", line 562, in min_time
    return keyfunc(min(vals, key=keyfunc))
ValueError: min() arg is an empty sequence

When using FunkLoad with a number of clients greater than 90, I get the following error :

Traceback (most recent call last):
   File "D:\funkload\lib\site-packages\funkload\FunkLoadTestCase.py", line 202, in _connect
    cert_file=self._certfile_path, method=rtype)
   File "D:\funkload\lib\site-packages\funkload\PatchWebunit.py", line 360, in WF_fetch
    h.endheaders()
   File "c:\Python27\Lib\httplib.py", line 937, in endheaders
    self._send_output(message_body)
   File "c:\Python27\Lib\httplib.py", line 797, in _send_output
    self.send(msg)
   File "c:\Python27\Lib\httplib.py", line 759, in send
    self.connect()
   File "c:\Python27\Lib\httplib.py", line 740, in connect
    self.timeout, self.source_address)
   File "c:\Python27\Lib\socket.py", line 571, in create_connection
    raise err
 error: [Errno 10048] Une seule utilisation de chaque adresse de socket (protocole/adresse réseau/port) est habituellement autorisée

This error in apparently caused by "TCP/IP port exhaustion", Windows keeps used sockets in TIME_WAIT state for 4 minutes after there have been used.

One solution is to use setsockopt on the socket to change the linger-on-close timeout value

socket.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, 1)

My (dirty) solution was to patch httplib.py, and after each

sock = socket.create_connection((self.host, self.port),
                                            self.timeout, self.source_address)

I added

sock.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, 1)

Is there a solution which would be better integrated with funkload and wouldn't need to hack into core python ?

Hi,

Funkload looks promising to me but. I followed the installation instructions (basically easy_install funkload) and believe that I have everything correctly installed. However when I run fl-install-demo then I get below mentioned error :

I am using Mac OS X 10.6.7

I could not find a place to ask this question so posting this in the issue section

Extract FunkLoad examples into ./funkload-demo : ...
Traceback (most recent call last):
File "/usr/local/bin/fl-install-demo", line 8, in
load_entry_point('funkload==1.15.0', 'console_scripts', 'fl-install-demo')()
File "build/bdist.macosx-10.6-i386/egg/funkload/DemoInstaller.py", line 11, in main
File "/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/pkg_resources.py", line 835, in resource_filename
self, resource_name
File "/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/pkg_resources.py", line 1305, in get_resource_filename
return self._extract_resource(manager, zip_path)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/pkg_resources.py", line 1316, in _extract_resource
zip_stat = self.zipinfo[zip_path]
KeyError: 'funkload/demo'

Regards & thanks
Kapil

In order to secure distributed tests, I propose to add LXC support.

• the prerequest are
  ** lxc is installed on the nodes
  ** lxc is properly configured to protect the system and the network. For example the funkload test should not be able to call a non authorized host
• the ssh call to run the test will start in an lxc container

I guess one way to do this is to create a LXCSSHDistributor class with some LXC options and have all SSH calls run into a LXC container

Chapter "First Steps with FunkLoad" refers to a Makefile, yet none exists.
The part in question is at the very end of tutorial.txt:

Configure the monitoring_ and automate your benchmark using a Makefile_.

when running distributed

• [ziade.org] returned
  [ziade.org]: bash: bin/fl-run-bench: Aucun fichier ou répertoire de ce type
  Traceback (most recent call last):
  File "bin/fl-run-bench", line 9, in
  load_entry_point('funkload==1.16.0', 'console_scripts', 'fl-run-bench')()
  File "/home/tarek/dev/hg.mozilla.org/funkload-demo/lib/python2.6/site-packages/funkload/BenchRunner.py", line 707, in main
  ret = distmgr.run()
  File "/home/tarek/dev/hg.mozilla.org/funkload-demo/lib/python2.6/site-packages/funkload/Distributed.py", line 469, in run
  self.logr_close()
  AttributeError: 'DistributionMgr' object has no attribute 'logr_close'

The ez_setup.py file seems to exist in the github repo, but not in the released packages. See traceback:

(funk)host-2-69:funk rob$ easy_install -f http://funkload.nuxeo.org/snapshots/ -U funkload
install_dir /Users/rob/tmp/funk/lib/python2.7/site-packages/
Searching for funkload
Reading http://funkload.nuxeo.org/snapshots/
Reading http://pypi.python.org/simple/funkload/
Reading http://funkload.nuxeo.org/
Best match: funkload 1.15.0
Downloading http://pypi.python.org/packages/source/f/funkload/funkload-1.15.0.tar.gz#md5=22748ba0b19907314b888ab4476eaf98
Processing funkload-1.15.0.tar.gz
Running funkload-1.15.0/setup.py -q bdist_egg --dist-dir /var/folders/Yu/YuJ3Jx+HEVq8QNoeOZVryk+++TI/-Tmp-/easy_install-TeFoGb/funkload-1.15.0/egg-dist-tmp-EPXGlF
Traceback (most recent call last):
File "/Users/rob/tmp/funk/bin/easy_install", line 9, in
load_entry_point('distribute==0.6.14', 'console_scripts', 'easy_install')()
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 1855, in main
with_ei_usage(lambda:
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 1836, in with_ei_usage
return f()
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 1859, in
distclass=DistributionWithoutHelpCommands, **kw
File "/opt/python/parts/opt/lib/python2.7/distutils/core.py", line 152, in setup
dist.run_commands()
File "/opt/python/parts/opt/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/opt/python/parts/opt/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 342, in run
self.easy_install(spec, not self.no_deps)
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 582, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 612, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 802, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 1079, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 1068, in run_setup
run_setup(setup_script, args)
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/sandbox.py", line 29, in run_setup
lambda: execfile(
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/sandbox.py", line 70, in run
return func()
File "/Users/rob/tmp/funk/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/sandbox.py", line 31, in
{'file':setup_script, 'name':'main'}
File "setup.py", line 24, in
ImportError: No module named ez_setup

When trying to install funkload via easy_install I ran into the following error:

Searching for funkload
Reading http://pypi.python.org/simple/funkload/
Reading http://funkload.nuxeo.org/
Best match: funkload 1.16.0
Downloading http://funkload.nuxeo.org/funkload-1.16.0.tar.gz
error: Can't download http://funkload.nuxeo.org/funkload-1.16.0.tar.gz: 404 Not Found

I've got following error:

$ fl-build-report --org bench_result.xml
Traceback (most recent call last):
  File "/home/k/build/fl-build-report", line 7, in 
    execfile(__file__)
  File "/home/k/funkload-repo/scripts/fl-build-report", line 25, in 
    main()
  File "/home/k/funkload-repo/src/funkload/ReportBuilder.py", line 295, in main
    options))
  File "/home/k/funkload-repo/src/funkload/ReportRenderOrg.py", line 79, in __init__
    RenderRst.__init__(self, config, stats, error, monitor, options)
TypeError: __init__() takes exactly 7 arguments (6 given)

Seems like change like this is needed:

diff --git a/src/funkload/ReportBuilder.py b/src/funkload/ReportBuilder.py
index 71becf9..7562c4c 100644
--- a/src/funkload/ReportBuilder.py
+++ b/src/funkload/ReportBuilder.py
@@ -292,7 +292,7 @@ def main():
             from ReportRenderOrg import RenderOrg
             print str(RenderOrg(xml_parser.config, xml_parser.stats,
                                 xml_parser.error, xml_parser.monitor,
-                                options))
+                                xml_parser.monitorconfig, options))
         else:
             print str(RenderRst(xml_parser.config, xml_parser.stats,
                                 xml_parser.error, xml_parser.monitor,
diff --git a/src/funkload/ReportRenderOrg.py b/src/funkload/ReportRenderOrg.py
index 0e33094..91c5f00 100644
--- a/src/funkload/ReportRenderOrg.py
+++ b/src/funkload/ReportRenderOrg.py
@@ -74,9 +74,9 @@ class RenderOrg(RenderRst):
     # number of slowest requests to display
     slowest_items = 5
 
-    def __init__(self, config, stats, error, monitor, options):
+    def __init__(self, config, stats, error, monitor, monitorconfig, options):
         options.html = True
-        RenderRst.__init__(self, config, stats, error, monitor, options)
+        RenderRst.__init__(self, config, stats, error, monitor, monitorconfig, options)
 
     def renderHeader(self):
         config = self.config

Like locust.io, I think Funkload would benefit from running Virtual users using Greenlets instead of Threads, using gevemt - this would allow each node to be able to run much more concurrent requests.

It could be a new option when running the bench runner - and threads would be kept as the default mode.

Hi,

In a staging process, we would like to test a different host that one registered in DNS.

Adding host via addHeader() method has no effect since _browser adds already host...

Example:

self.debugHeaders(debug_headers=True) 
self.setHeader('host', 'www.myhost.tld')
self.get(server_url+'/')

Displays:

Putting header -- Host: www.realhost.tld
Putting header -- User-Agent: FunkLoad/1.16.1
Putting header -- host: www.myhost.tld

I think it's a common issue.

Thx for your work.

In a large test in distributed mode a large amount of data is transferred back. Currently the stopmonitors occurs after this data transfer which means that monitor graphs include data on the transfer as well as the testing.
The stopmonitor should occur first and then all the data transfer, including that of the stats themselves.

Would be great if the html benchmark report could use 'set terminal canvas' or other configurable terminal output types.

when running the bench distributed, we should be able to send a SIGTERM to the funkload script so it aborts the run, exactly like the ctrl-c does.

right now, kill the process will let the ssh session open and hanging..

will try to provide a patch

Hi,

Starting to use this tool and have some questions. Some how feel uncomfortable posting these questions in the "Issues" section and can not find anywhere any forum or google group.

My humble suggestion is to create a forum where the experts can answer users' queries.

If it exists please pardon my ignorance and point me towards it.

Regards & thanks
Kapil

Unittest in Python 2.7 does not include _exc_info(), which means as people upgrade to the new version of Ubuntu Linux you will increasingly see complaints and questions relating to the non-existence of this code, instead of funkload users seeing their previously working test code's test failure information.

Adding the following code:

def _exc_info(self):
    return sys.exc_info()

to FunkLoadTestCase fixed this for me (I added it to my derived classes).

Please see here: http://www.gossamer-threads.com/lists/python/python/732128 where Python's unittest.py replaced _exc_info() with sys.exc_info().

 James

We should be able to configure log_path fro the command line since it's a directory local to the master.

Here's what I had to do for Funkload working with deleting cookies in my case:

diff --git a/src/funkload/PatchWebunit.py b/src/funkload/PatchWebunit.py
index b780b64..ce85eb6 100644
--- a/src/funkload/PatchWebunit.py
+++ b/src/funkload/PatchWebunit.py
@@ -337,7 +337,7 @@ def WF_fetch(self, url, postdata=None, server=None, port=None, protocol=None,
                 # TODO: check for expires (max-age is working)
                 # hard coded value that application can use to work
                 # around expires
-                if sendcookie.coded_value in ('"deleted"', "null"):
+                if sendcookie.coded_value in ('"deleted"', "null", "deleted"):
                     continue
                 cookie_list.append("%s=%s;"%(sendcookie.key,
                                             sendcookie.coded_value))

I'm not sure if it's a typo with first "deleted" (double quote inside of single quote) or it's like this by purpose. Anyway, i had to add "deleted" without additional quotation marks for me to work. I would be glad if you could add this simple fix to mainstream version.

I've got following error:

$ fl-build-report --html bench_result.xml
Creating html report: ...Traceback (most recent call last):
  File "/home/k/build/fl-build-report", line 7, in 
    execfile(__file__)
  File "/home/k/funkload-repo/scripts/fl-build-report", line 25, in 
    main()
  File "/home/k/funkload-repo/src/funkload/ReportBuilder.py", line 288, in main
    xml_parser.monitorconfig, options)()
  File "/home/k/funkload-repo/src/funkload/ReportRenderHtmlBase.py", line 126, in render
    self.createRstFile()
  File "/home/k/funkload-repo/src/funkload/ReportRenderHtmlBase.py", line 85, in createRstFile
    f.write(str(self))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u017a' in position 5251: ordinal not in range(128)

Here's my solution:

diff --git a/src/funkload/ReportBuilder.py b/src/funkload/ReportBuilder.py
index 71becf9..3f3a9d9 100644
--- a/src/funkload/ReportBuilder.py
+++ b/src/funkload/ReportBuilder.py
@@ -290,13 +290,13 @@ def main():
             trace(html_path + "\n")
         elif options.org:
             from ReportRenderOrg import RenderOrg
-            print str(RenderOrg(xml_parser.config, xml_parser.stats,
+            print unicode(RenderOrg(xml_parser.config, xml_parser.stats,
                                 xml_parser.error, xml_parser.monitor,
-                                options))
+                                xml_parser.monitorconfig, options)).encode("utf-8")
         else:
-            print str(RenderRst(xml_parser.config, xml_parser.stats,
+            print unicode(RenderRst(xml_parser.config, xml_parser.stats,
                                 xml_parser.error, xml_parser.monitor,
-                                xml_parser.monitorconfig, options))
+                                xml_parser.monitorconfig, options)).encode("utf-8")
 
 
 if __name__ == '__main__':
diff --git a/src/funkload/ReportRenderHtmlBase.py b/src/funkload/ReportRenderHtmlBase.py
index 92ff9f3..00b67d0 100644
--- a/src/funkload/ReportRenderHtmlBase.py
+++ b/src/funkload/ReportRenderHtmlBase.py
@@ -82,7 +82,7 @@ class RenderHtmlBase(RenderRst):
         """Create the ReST file."""
         rst_path = os.path.join(self.report_dir, 'index.rst')
         f = open(rst_path, 'w')
-        f.write(str(self))
+        f.write(unicode(self).encode("utf-8"))
         f.close()
         self.rst_path = rst_path

There may be some more places that need to be changed like this but for now, this is enough for me.

As far as I can tell, currently distribute mode only supports ssh auth via public key or user+password. Are there plans to allow specification of private key too (main use case: EC2 boxes)?

Thanks!

It would be nice to display a merged feedback when doing distributed tests

It's hard to deeply customize the layout of some reports like the trend report, without changing the code.

This is just an idea, as it seems to change a lot of internals: what about moving the layout of the reports (the reST layout I guess) to a template file, using mako or something equivalent.

That would allow the user to do everything and anything by providing a custom template, without touching the code.

My last test generated 600M of data which is significant. This could be significantly reduced if the log files were gziped before being retrieved. The monitor server can be made to compress its data before sending too.
It would be even better if fl-build-report to read gziped log data to save space on disk.

Hi, I'm using Python 2.7.1 and FunkLoad 1.16.1, and get invalid XML when running the tests. Here's an example output:

wei:simple wei$ make
fl-run-test test_Simple.py -v 
test_simple (test_Simple.Simple) ... Ok

----------------------------------------------------------------------
Ran 1 test in 0.041s

OK
wei:simple wei$ fl-build-report simple-test.xml 
Error: invalid xml bench result file
Note that you can generate a report only for a
                    bench result done with fl-run-bench (and not on a test
                    resu1lt done with fl-run-test).
Xml parser element stack: ['root']
Traceback (most recent call last):
  File "/usr/local/bin/fl-build-report", line 8, in <module>
    load_entry_point('funkload==1.16.1', 'console_scripts', 'fl-build-report')()
  File "/Library/Python/2.7/site-packages/funkload-1.16.1-py2.7.egg/funkload/ReportBuilder.py", line 285, in main
  File "/Library/Python/2.7/site-packages/funkload-1.16.1-py2.7.egg/funkload/ReportBuilder.py", line 104, in parse
xml.parsers.expat.ExpatError: junk after document element: line 2, column 0

The generated XML is here: https://gist.github.com/1a3d55e8a386bbf514ad

Hello,

I am testing an SSL based web application. I use self signed certificates
and everything works well. I am trying to write an test with help of Funkload, so I use the

setKeyAndCertificateFile(keyfile_path, certfile_path)

method. The code looks something like this:

def setUp(self):
"""Setting up test."""
self.logd("setUp")
self.server_url = self.conf_get('main', 'url')
self.setKeyAndCertificateFile('client.key','client.crt')

The *.key and *.crt file were created with openssl and were signed with ca.crt (which
also I created with openssl). The certs are pem formated. I instructed the apache server to trust certificates signed with ca.crt.

As I mentioned in browser everything works fine, but with Funkload I get an ssl handshake error exception.

SSLError: [Errno 1] _ssl.c:480: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

Is this an issue or am I missing something?

Hi all,

i want to run the funkload monitoring server on a CentOS 5.8 System. I installed the funkload minimum as described in the documentation. Whe i start the Server i get the following error.

fl-monitor-ctl monitor.conf start

Traceback (most recent call last):
File "/usr/bin/fl-monitor-ctl", line 7, in ?
sys.exit(
File "/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r88846-py2.4.egg/pkg_resources.py", line 318, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r88846-py2.4.egg/pkg_resources.py", line 2221, in load_entry_point
return ep.load()
File "/usr/lib/python2.4/site-packages/setuptools-0.6c12dev_r88846-py2.4.egg/pkg_resources.py", line 1954, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "build/bdist.linux-x86_64/egg/funkload/Monitor.py", line 27, in ?
File "/usr/lib/python2.4/site-packages/funkload-1.16.1-py2.4.egg/funkload/MonitorPlugins.py", line 28
class MonitorPlugins():
^
SyntaxError: invalid syntax

I have no idear what to do. ;)

Thx

In the monitor plugin there is an assertion for kernel version between 2.4 and 2.6 this should be removed.

I would like to have a number of different configuration files, and specify which to use on the command line, rather than have it tied to the class name. Am I misunderstanding how people use FunkLoad? How do you run the same test in different configurations?

I'd be glad to work up a patch if you're open to the idea.

I'm merging results using fl-build-report logs/stats.xml logs/node1.xml ...

This results in a report that includes monitor stats except for 2 bugs.

Monitor data is considered a node so increases node count which leads to incorrect concurrent users

The CU's graph on the monitor hosts seems to show a single value for all time values instead of the correct CU's

When running funkload benchmarks overnight, I've noticed that at midnight, all of the hardware graphs wraparound, creating a lot of noise in the process. It also interferes with the scale of the plot, making the graph much harder to read.

It appears that gnuplot is ordering benchmark results by time of day, rather than by date/time.

Hi,

FunkLoad is a pretty god tool to run functional tests on WebFrameworks, but you can improve the load you create on a web-server, I don't think that you can saturate a server written in C with Python, except you add the feature to run distributed load tests. Or you make your internal load generator replaceable with external tools, like weighttp for example.

I installed tcpwatch 1.3.1 using pip and ended up with an executable called simply tcpwatch. Can that be added into the list in Recorder.py along with tcpwatch-httpproxy and tcpwatch.py? I know I can work around this by setting the TCPWATCH environment var but it would be better to not have to.

When I tried to run fl-install-demo I got the following error:
OSError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/funkload-1.15.0-py2.7.egg/funkload/demo'

The demos folder for me existed at '/usr/share/doc/funkload/examples/demo' while the script was looking under the egg folder. A patch for me was the same as documented at the following URL:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=630236

(TornadoTest)[mp262554@khaki01:~/TornadoTest]$ bin/fl-run-bench test_simple.py Simple.test_simple --distribute
No handlers could be found for logger "FunkLoad"

When specifying usernames and passwords for the host, I expect the format to be the same on both the command line on the config. I.e., on the command line:

fl-run-bench --distribute [email protected]

...will be parsed out correctly (username=amcfague, host=github.com). When specifying the same line in the config file, it treats the entire string as the hostname. While the config file DOES allow you to specify a global username for all hosts, it doesn't make sense to circumvent the normal SSH URI.

Just found Funkload use HTTP 1.0 by default, is there anyway to use HTTP 1.1?

[UPDATE] This is because webunit.webunittest.py and PatchWebunit.py use obsolete httplib.HTTP(which is retained only for backward compatibility with 1.5) instead of httplib.HTTPConnection.

So is there any impact on the testing result?

Anyone could give more details, thanks.

Hi guys,

I'm just starting with FunkLoad, and it seems wonderful. Just one thing that popped up when I started testing our Django-based app:

The docs at http://funkload.nuxeo.org/recorder.html say that fl-record "handles automaticly JSF Myfaces token, which enable to easily record and play any JBoss Seam application".

Is similar support for the Django CSRF tokens in the works?

Thanks,
Shai.

I don't really know if this is an unimplemented feature or is this a bug.

The problem that I'm having is that although the underlying library does record cookie information, FunkLoad fl-record seems to totally ignore them. Is this an intended behaviour? If so, where could I start to implement it properly. I know there is a way to add "Cookie" head parameter just like any other.

Thanks for the awesome tool!

If I run distributed from a virtualenv, I get this error (until I install virtualenv in the virtualenv -- which is silly)

$ bin/fl-run-bench --distribute --distribute-workers=ziade.org test_simple.py Simple.test_simple

No handlers could be found for logger "FunkLoad"

Benching Simple.test_simple

Access our Demo app

Configuration

• Current time: 2011-07-27T12:50:06.545231
• Configuration file: /home/tarek/dev/hg.mozilla.org/funkload-demo/Simple.conf
• Distributed output: distributed-simple-test.log
• Server: http://localhost:5000
• Cycles: [5, 10, 20]
• Cycle duration: 10s
• Sleeptime between request: from 0.0s to 0.5s
• Sleeptime between test case: 0.01s
• Startup delay between thread: 0.01s
• Workers :ziade.org
• Preparing sandboxes for 1 workers.Exception in thread Thread-3:
  Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
  self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
  self.__target(_self.__args, *_self.__kwargs)
  File "/home/tarek/dev/hg.mozilla.org/funkload-demo/lib/python2.6/site-packages/funkload/Distributed.py", line 372, in local_prep_worker
  get_virtualenv_script(),
  File "/home/tarek/dev/hg.mozilla.org/funkload-demo/lib/python2.6/site-packages/funkload/utils.py", line 314, in get_virtualenv_script
  pkg = pkg_resources.get_distribution('virtualenv')
  File "/home/tarek/dev/hg.mozilla.org/funkload-demo/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/pkg_resources.py", line 292, in get_distribution
  if isinstance(dist,Requirement): dist = get_provider(dist)
  File "/home/tarek/dev/hg.mozilla.org/funkload-demo/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/pkg_resources.py", line 176, in get_provider
  return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/home/tarek/dev/hg.mozilla.org/funkload-demo/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/pkg_resources.py", line 648, in require
  needed = self.resolve(parse_requirements(requirements))
  File "/home/tarek/dev/hg.mozilla.org/funkload-demo/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/pkg_resources.py", line 546, in resolve
  raise DistributionNotFound(req)
  DistributionNotFound: virtualenv
• Starting 1 workers.
• [ziade.org] returned
  [ziade.org]: bash: bin/fl-run-bench: Aucun fichier ou répertoire de ce type
  Traceback (most recent call last):
  File "bin/fl-run-bench", line 9, in
  load_entry_point('funkload==1.16.0', 'console_scripts', 'fl-run-bench')()
  File "/home/tarek/dev/hg.mozilla.org/funkload-demo/lib/python2.6/site-packages/funkload/BenchRunner.py", line 707, in main
  ret = distmgr.run()
  File "/home/tarek/dev/hg.mozilla.org/funkload-demo/lib/python2.6/site-packages/funkload/Distributed.py", line 469, in run
  self.logr_close()
  AttributeError: 'DistributionMgr' object has no attribute 'logr_close'

Hi,

We use a somewhat lengthy scenario (takes ~150s); since the logging phase seems to always wait the full length of the cycle as defined, I was wary of using the recommended "5X" for cycle time and used 300 instead.

When the number of threads rises -- and not to crazy sizes, but only to 20-30 -- I see issues where Funkload fails to collect all threads. Please see log below (I removed some identifying details, but it still should be clear) -- an example where, after successfully running through the scenario with 10, 20, and then 30 threads, fl-bench just hangs, seemingly under the impression that some threads have not completed yet.

I will appreciate any hints on this, and try to provide whatever further info in my power. This is FunkLoad 0.16.1 (installed by pip in a virtualenv) on Python "2.7.1+" (whatever that means... it's the system's Python) on Ubuntu 11.04.

========================================================================
Benching HomePage.test_home_page
========================================================================
XXX the test case description
------------------------------------------------------------------------

Configuration
=============

* Current time: 2011-10-03T15:24:36.491456
* Configuration file: .../HomePage.conf
* Log xml: .../home_page-bench.xml
* Server: http://...com
* Cycles: [10, 20, 30]
* Cycle duration: 300s
* Sleeptime between request: from 0.0s to 2.0s
* Sleeptime between test case: 1.0s
* Startup delay between thread: 0.2s

Benching
========

* setUpBench hook: ... done.

Cycle #0 with 10 virtual users
------------------------------

* setUpCycle hook: ... done.
* Current time: 2011-10-03T15:24:36.495190
* Starting threads: .......... done.
* Logging for 300s (until 2011-10-03T15:29:38.533322): .......... done.
* Waiting end of threads: .......... done.
* Waiting cycle sleeptime 1s: ... done.
* tearDownCycle hook: ... done.
* End of cycle, 359.64s elapsed.
* Cycle result: **SUCCESSFUL**, 10 success, 0 failure, 0 errors.

Cycle #1 with 20 virtual users
------------------------------

* setUpCycle hook: ... done.
* Current time: 2011-10-03T15:30:36.131613
* Starting threads: .................... done.
* Logging for 300s (until 2011-10-03T15:35:40.209317): .................... done.
* Waiting end of threads: .................... done.
* Waiting cycle sleeptime 1s: ... done.
* tearDownCycle hook: ... done.
* End of cycle, 389.49s elapsed.
* Cycle result: **SUCCESSFUL**, 20 success, 0 failure, 0 errors.

Cycle #2 with 30 virtual users
------------------------------

* setUpCycle hook: ... done.
* Current time: 2011-10-03T15:37:05.617982
* Starting threads: .............................. done.
* Logging for 300s (until 2011-10-03T15:42:11.747434): ............................. done.
* Waiting end of threads: ..........................^Z
[1]+  Stopped                 fl-run-bench -c10:20:30 --accept-invalid-links test_HomePage.py HomePage.test_home_page
(funkload)$ fg
fl-run-bench -c10:20:30 --accept-invalid-links test_HomePage.py HomePage.test_home_page
User defined signal 1
(funkload)$ date
Mon Oct  3 16:10:37 IST 2011

(the "date" invocation in the end is intended to show that I was patient... the set of successful tests ended by 15:42, but fl-bench was still waiting for threads half an hour later. Even then, I tried to gently "nudge" it into proper ending by kill -USR1, but that didn't help much).

RecorderProgram.extractRequests currently contains the following lines:

    filter_ctypes = ('image', 'css', 'javascript')
    filter_url = ('.jpg', '.png', '.gif', '.css', '.js')

Could we have 'x-shockwave-flash' added to filter_ctypes, and '.swf' added to filter_url?

I've got the following error:

python setup.py install
Traceback (most recent call last):
File "setup.py", line 24, in
import ez_setup
ImportError: No module named ez_setup

After manual ez_setup installation I've got no problems.

You should add 'ez_setup' to 'install_requires' in setup.py

In theory using async io should result in being able to handle many more concurrent connections that python threads.
For example Locust uses this (https://github.com/cgbystrom/locust) and Tsung.
To test this out I integrated gevent by adding the following line to the top of BenchRunner.py:

  from gevent import monkey; monkey.patch_all()

The result was a 40-50% decrease in cpu usage however the stats recorded weren't the same. Most dramatic was the STPS dropped from 120 down to 75. I think something changed in how the killing of the thread occurred. It looks like the gevent ran only 43 tests whereas without ran 76 tests in the same 60 duration. Generally longer request duration were recorded but the rps was similar.

More investigation is needed.

In large tests, the "Page detail stats" can grow to enormous sizes.

It'd be nice to have a flag that can be passed to fl-build-report to leave this section out.

What is the best way to loadtest a site with mutliple URLS ( ~10000 urls) in a distributed fashion ? I was thinking of writing a Loadtest Case that opens the file ( file containing the urls) and randomly choosing a URL. Is this a suggested way ?

While running a simple test in distributed mode with the following command:

where localhost is mac osx 10.7 and frankie.local is ubuntu 11.04, both with python2.7

I encountered two errors,

• rmdir -rf on the sandbox directory failed
• easy_install {funkload} failed.

Under Centos5 I needed to install a custom version of gnuplot, so the executable is in /usr/local/bin

Funkload uses "gnuplot", which works as long as the PATHs are set and the binary has the same name.

To ease this, what about adding a --with-gnuplot=/path/to/gnuplot option to fl-build-report ?

Hi, I install funkload via easy_isntall onto Fedora 14. It seems to work, although the first strange thing I noticed is that the demo directory was missing. No big deal. I then ran a bench with seemingly no problems, but trying to generate html from the xml it gives me

Creating html report: ...Traceback (most recent call last):
  File "/usr/bin/fl-build-report", line 9, in <module>
    load_entry_point('funkload==1.15.0', 'console_scripts', 'fl-build-report')()
  File "/usr/lib/python2.7/site-packages/funkload-1.15.0-py2.7.egg/funkload  /ReportBuilder.py", line 283, in main
    options)()
  File "/usr/lib/python2.7/site-packages/funkload-1.15.0-py2.7.egg/funkload/ReportRenderHtmlBase.py", line 127, in render
    self.copyCss()
  File "/usr/lib/python2.7/site-packages/funkload-1.15.0-py2.7.egg/funkload/ReportRenderHtmlBase.py", line 98, in copyCss
    css_content = resource_string('funkload', 'data/funkload.css')
  File "/usr/lib/python2.7/site-packages/distribute-0.6.15-py2.7.egg/pkg_resources.py", line 881, in resource_string
   self, resource_name
  File "/usr/lib/python2.7/site-packages/distribute-0.6.15-py2.7.egg/pkg_resources.py", line 1154, in get_resource_string
    return self._get(self._fn(self.module_path, resource_name))
  File "/usr/lib/python2.7/site-packages/distribute-0.6.15-py2.7.egg/pkg_resources.py", line 1281, in _get
    stream = open(path, 'rb')
IOError: [Errno 2] No such file or directory: '/usr/lib/python2.7/site-packages/funkload-1.15.0-py2.7.egg/funkload/data/funkload.css

The data directory does not exist at all, and I cannot find funkload.css anywhere else on the system.

unittest.TestCase._exc_info is not available in Python 2.7 which causes addFailure calls to fail when running FunkLoad tests on Python 2.7.

I think the fix is similar to the patch at the end of this post: http://www.mail-archive.com/[email protected]/msg00680.html

Cheers,

Jim

In addition to the main thread, there were threads (one per node) that were hanging around, preventing the return of main() in BenchRunner.py from completing. I added a line to kill all the workers and that did the trick:

...
if options.distribute:
    from Distributed import DistributionMgr
    ret = None
    try:
        distmgr = DistributionMgr(
            args[0], klass, method, options, cmd_args)
    except UserWarning, error:
        trace(red_str("Distribution failed with:%s \n" % (error)))

    try:
        distmgr.prepare_workers(allow_errors=True)
        ret = distmgr.run()
        distmgr.final_collect()
    except KeyboardInterrupt:
        trace("* ^C received *")
        distmgr.abort()

     # added this line
     [w.die() for w in distmgr._workers]
     return ret
else:
     ...