Home > FOSS, ownCloud > DAV Torture

DAV Torture

Currently we speak a lot about performance of the ownCloud WebDAV server. Speaking with a computer programmer about performance is like speaking with a doctor about pain. It needs to be qualified, the pain, and also the performance concerns.

To do a step into that direction, here is a little script collection for you to play with if you like: the DAV torture collection. We started it quite some time ago but never really introduced it. It is still very rough.

What it does

The first idea is that we need a reproducable set of files to test the server with. We don’t want to send around huge tarballs with files, so Danimo invented two perl scripts called torture_gen_layout.pl and torture_create_files.pl. With torture_gen_layout.pl one can create a file that contains the layout of the test file tree, a so called layout( or .lay)-file. The .lay-file describes the test file tree completely, with names, structure and size.

torture_gen_layout.pl takes the .lay-file and really creates the file tree on a machine. The cool thing about is that we can commit on a .lay-file as our standard test tree and just pass a file around with a couple of kbytes size that describes
the tree.

Now that there is a standard file tree to test with, I wrote a little script called dav_torture.pl. It copies the whole tree described by a .lay file and created on the local file system to an ownCloud WebDAV server using PUT requests. Along with that, it produces performance relevant output.

Try it

Download the tarball and unpack it, or clone it from github.

After having installed a couple of perl deps (probably only modules Data::Random::WordList, HTTP::DAV, HTTP::Request::Common are not in perl’s core) you should be able to run the scripts from within the directory.

First, you need to create a config file. For that, copy t1.cfg.in to t1.cfg (don’t ask about the name) and edit it. For this example, we only need user, passwd and url to access ownCloud. Be careful with the syntax, it gets sourced into a perl script.

Now, create the local reference tree with a .lay-file which I put into the tarball:

./torture_create_files.pl small.lay tree

This command will build the file tree described by small.lay into the directory called tree.

Now, you can already treat your server: Call

./dav_torture.pl small.lay tree

This will perform PUT commands to the WebDAV server and output some useful information.
It also appends to two files results.dat and puts.tsv. results.dat just logs the results of subseqent call. The tsv file is the data file for the html file index.html in the same directory. That opened in a browser gives a curve over the average transmission rate of all subsequent runs of dav_torture.pl (You have run dav_torture.pl a couple of times to make that visible). The dav_torture.pl script can now be hooked into our Jenkins CI and performed after every server checkin. The resulting curve must never raise :-)

To create your own .lay-file, open torture_gen_layout.pl and play with the variables on top of the script. Simply call the script and redirect into a file to create a .lay-file.

All this is pretty experimental, but I thought it will help us to get to a more objective discussion about performance. I wanted to open this up in a pretty early stage because I am hoping that this might be interesting for somebody of you: Treat your own server, create interesting .lay files or improve the script set (testing plain PUTs is rather boring) or the result html presentation.

What do you think?

About these ads
Categories: FOSS, ownCloud Tags: , , ,
  1. September 27, 2013 at 17:47

    Cool! You should post this to the SabreDAV list http://groups.google.com/group/sabredav-discuss

    • dragotin
      September 27, 2013 at 20:39

      Will do once it is a bit more mature.

  2. Diederik
    September 27, 2013 at 23:09

    Share it on github, so everyone can submit pull request to improve it. It also makes it easier to view/’share’ it. And +1 on Thomas’ suggestion, even in this (apparently) unmature stage. The SabreDAV maintainer has a rather cooperative attitude and I think he’ll appreciate it.

  3. dragotin
    September 29, 2013 at 21:34

    It is now checked in here:

    git@github.com:owncloud/administration.git
  4. October 4, 2013 at 15:10

    For installing this on a raspberry pi you need some more stuff:
    enter in terminal:

    sudo apt-get install perl curl cpp expat libssl-dev libxml-parser-perl
    curl -L http://cpanmin.us | perl – –sudo App::cpanminus
    sudo cpanm Carp::Assert
    sudo cpanm Data::Random::WordList
    sudo cpanm HTTP::DAV
    sudo cpanm HTTP::Request::Common
    sudo cpanm LWP::Protocol::https

    you need the https protocol if your owncloud instance is https secured. (what it should be concerning newest developments in internet controlling)

  5. Martin
    October 6, 2013 at 22:20

    Hello,

    I get the following error:
    [...]
    Creating local dir: turbo/
    Operation failed. You can only open a collection (directory)
    Failed to create directory : 0

    ?
    I can see a connect to the server in my web server’s log file an I have set:

    ld_libpath => “/usr/lib/x86_64-linux-gnu/ocsync-0″,
    csync => “/usr/bin/ocsync”
    in t1.cfg (on a Debian system)

    Thanks & best regards

    Martin

  6. etiess
    October 9, 2013 at 12:23

    Hello,

    Is this work related to https://github.com/owncloud/mirall/issues/331 and/or https://github.com/owncloud/core/issues/3118 ? Is mirall using WebDAV too?

    Thanks for this work: im my opinion, this is much more important than new features (I’m suprized that new features are announced, while owncloud is not usable in a productive environment: https://twitter.com/fkarlitschek/status/386047838146162688).

    • dragotin
      October 9, 2013 at 13:05

      Yes, mirall uses WebDAV so all improvements we’re doing in that area automatically make mirall faster.

      • etiess
        October 9, 2013 at 13:12

        Great! And sorry for not helping a lot there, but this is unfortunately to complicated for me :(

        Bon courage!

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: