Archive

Posts Tagged ‘csync’

Csync Upstream Release 0.50.0

August 5, 2013 1 comment

Last week Andreas did an upstream release of the file synchronization software csync. Frequent readers know that csync is the sync engine that is used in the ownCloud client, so this is a very important and special release for us.

Yeah for upstream! The new release contains a lot of features and changes me and my collegues worked on during the last 18 month: First we added the ownCloud module to csync upstream, so that csync now is able to sync local directories to an ownCloud server. The ownCloud client works on platforms Linux and MacOSX and Windows. That required a lot of (tricky) changes to the csync platform which we carefully backported to csync upstream.

There are much more changes, such as: More compilers supported, a new logging framework, a new base lib to do unit testing, removal of not so common dependencies and other infrastructure changes. But also on the feature side there are more improvements, mainly to focus on a more easy and broader use of the csync library that allows to embed the sync functionality to various backends into other applications.

I am very happy that we kept our promise to contribute a lot of our changes back to upstream. Primarily that is the way to go in the open source world, you will say. But it was distressing to see how quickly people were whispering stuff like “Well, wouldn’t it be easier if you do a fork? Remember, you have to come to good results quickly for the company, why bother with an upstream project?”

For me, that wouldn’t be like companies in FOSS world should behave, and I know that on the long run, it would not be beneficial for the company either. So very good that the relevant people supported the idea of going with upstream from the beginning.

Does that mean that csync 0.50.0 and ocsync as shipped with ownCloud are the same now? No, unfortunately not. We did more changes on the ownCloud branch which need to be reviewed and probably won’t fit to csync the way they are now.
More work to be done, but we’re on the way!

Categories: FOSS, ownCloud, Release Tags: , ,

Sync Progress Display

July 23, 2013 16 comments

here is something new and eye candy in the ownCloud Client, so let me show a bit of what we have worked on recently.

Many users of the ownCloud Client were asking for sync progress information, in fact there was none at all until today which is a bit boring. The reason why we hadn’t it was simply that csync, which is the file synchronizer engine we use, did not have an API to hand over progress information of an actual up- or download to higher levels of the application.

We implemented two callbacks in csync: One that informs about start, end and progress of an up- or download of an individual file. Another one processes the overall progress of the currently running sync run, indicating for example that eight files have to be processed, current is file number four, and x of the overall sum y bytes have been processed already.

That information is passed into a singleton class called ProgressDispatcher in the client code, where other classes can connect to a signal providing that information. That comes handy as we need the information at different places in the client GUI.

Sync Progress Display

Sync Progress Display


The screenshot shows the first and probably most detailed implementation of the progress display in the sync accounts details which are part of a new settings- and status dialog.

The visual appearance was worked out by our interaction designer Jan. We always have to argue a bit because obviously I am the greatest interaction designer around 😉 but well, finally we do like Jan says and the result is great IMO. It’s great that we have him for ownCloud as that guarantees that our project is not drifting too much onto the geeky techy side which (I heard) is scaring off some users…

Hope you like it! You can get a preview in the current nightly builds (win, mac, Linux) of the client! Please report bugs as you find them, thanks.

A week for csync

December 9, 2012 11 comments

On Friday I arrived back from Berlin where I had the pleasure to work with my great colleague Danimo and our friends from Woboq, Markus and Olivier, in the Woboq Headquarter in Berlin Kreuzberg for a week.

We thought that it might be fun to work together on csync, our sync engine under the hood of the ownCloud client. There were some issues that should be fixed and on the way we cleaned and improved quite some code in csync.

Here are some things we worked on:

  • We added a function that lets the program that uses the csync library pass arbitrary module parameters to the backend module. That way its more easy to steer the behaviour of the ownCloud modules from the calling app.
  • Error handling was improved, ie. if an http error happens, csync works errno based error reporting. We added custom errnos because not all error cases with http can be mapped to system errnos.
  • Formerly the csync ownCloud module was spooling files through an additional temporar file on client side. That step is skipped now which results in performance improvements as well as in more clean code.
  • We were able to reduce the number of HTTP requests that go over the wire even more. For example to check if there are changes on server side, now there is exactly one http propfind required. Also if files have to be synced, we could save some HTTP requests by improving caching of some requests.
  • Andreas recently changed the logging system in csync upstream master branch. We merged that back and now do not longer need the log4c framework. One build dependency less and a nice new logging framework.
  • Other bugs were fixed, such as a potential crash if a folder as deleted during it is synced, SSL handling shortcomings, code streamlining in handling compressed data streams and more.
  • We finalized a patch that uniforms the utf8 representations of characters over all platforms. That will fix problems we saw especially with MacOS and special filenames.

Ah, yes, we also did other things, more related to the ownCloud client. Danimo managed to implement a cross platform filesystem watcher class that is able to fulfill our requirements. That obsoletes polling for changes on the local file system, one of the most popular enhancement requests.

And finally there now is a API in csync thats reports file transmission progress if a callback is installed accordingly. So the client hopefully soon will tell ya what it’s doing for you. Also appreciated I guess…

Last but not least we added code to use QtKeyChain, a cross platform password storage library that stores password encrypted. For example on Linux QtKeyChain connects to kwallet. QtKeyChain was provided by
Frank Osterfeld, thanks a lot for that contribution.

Quite some stuff for a short week, note that stuff that fills a short line in this blog can be quite nifty to investigate, implement and test. Not everything is stable, polished and properly integrated but it was a great and productive week. The next release of ownCloud Client will be a nice one.

woboq_dinner2
And sice you can not always work, we had a nice dinner at a very cool italian restaurant. We met with other ownCloud employees located in Berlin, Arthur and Georg. Fun 🙂 And Berlin, yes, a great place to be, but finally I appreciated to arrive back to my snow covered home.

Many thanks to Olivier and Markus for hosting us and for the nice week.

Csync for ownCloud Client 1.1.0 – A New Sync Engine

October 11, 2012 18 comments

Along with todays ownCloud 4.5 release we released the new ownCloud Client 1.1.0 with a new syncing concept.

This blog will shed some light on the details. I apologize, it’s a long read.

Time Issues

ownCloud Client versions 1.0.x worked with csyncs traditional way of using the file modification times to detect updates between the two repositories that should be synced to each other. That works fine and conforms to our idea to ideally not use any other metadata in syncing than what the file system has anyway.

Time flies

However, there is one drawback which we all know from daily life: If at least two parties sync on time its important that all clocks are set exactly the same way. Remember good crime movies where a bank robbery always starts with a clock adjustment of all gangsters? We have exactly the same in ownClouds syncing: All involved have to have the same time setting, otherwise modification times of files can not be compared reliably.

There are solutions for computers to set the exact time (like ntp) so in general that works. However, in real life scenarios these are not reliable because either people do not have them started on the system or because the daemon updates the time once in a while and in that time span the clock skews already too much.

Users all the time reported problems with that and other experts continued to advise that we never get around that problems if we don’t change something fundamental and go away from pure time based syncing.

Well, we did that with our csync version 0.60.0 which is the sync engine for ownCloud Client 1.1.0.

An Unique Id

Now, every file and directory inside a sync directory has an unique Id associated. The idea is that the Id changes if the file changes. So in the sync process the need for a file update in either direction can be computed by comparing the two Ids of the file. If the id has changed on one repository the file was changed there and needs to be synced to the other side.

The Ids are generated on the ownCloud server and one challenge for the client is to always download the correct Id of a file. The Ids are just random tags for a file version. It is not associated to the file content as MD5 sums would be. Actually it was a frequent advise to use MD5 sums or a similar approach which digests the files content to detect updates. That would have come very handy because that means comparing file contents directly and, more important, it’s reproducable on either side. Also the client would have been able to recalculate the MD5-Sum of the local files and would not have depended on a local database with Ids that were pulled from the server before.

But we decided against hashes. Calculating MD5-Sums is costly in terms of CPU and time, especially for large files. The CPU problem is small on clients, but not on servers where a lot of clients connect to. Even though the sums can be calculated during upload, the problems remain for the case where the server does not see the upload stream, think of the “mount my Dropbox” case.

For files on the ownCloud server, the Id is always updated when the file gets updated. On the client side the last Id of a file is in the client database. It is invalidated in case the files modification time changed meanwhile to detect local changes.

Change Propagation

Another remarkable change in the 1.1.0 client is that change events in the file tree propagate up to the top directory on the owncloud server, ie. if a file changes in a directory, the id of the directory changes as well as the one of its parent directory etc.

That means that to detect if a file tree has changed, it’s enough to check the top most directories Id. If that has changed, ok, than the client needs to dig deeper, but in the not so rare case that nothing has changed, the one call is enough to detect that. That dramatically lowers the server load with clients because instead of digging through the whole directory structure what we did with the 1.0.x series
it is a few requests now.

CSync and ownCloud for Success

These are very intrusive changes to csync. For example, we had to add two additional fields to the database, add code that is able to build a representation
of the local file tree from the database and make csync query for the file Ids from
the server if needed. Deep under the hood the updater, reconciler and propagator code needed changes to work with the Ids. All these changes did not go back to csync upstream yet.

To not conflict with the upstream version of csync we decided to rename our csync version to ocsync. But: This is a temporar solution for the time we need to catch up with upstream again. That will take a while until everything is sorted again but we will work on that.

I am are very excited about the new version of csync. But obviously there are other changes in the ownCloud Client 1.1.0 which will be subject of another blog post.

CSync and Mirall Development Setup

March 22, 2012 51 comments

A KDE bagger

Build it!


people were asking how to set up a development setup for the syncing client we are working on to sync local files to ownCloud and vice versa currently, work title mirall. While a website about it is not yet finished, I try to summarize it here. There are some hacks here and there but that’s how I do it today. It will improve over time. Note that this is about a real development setup, not for a production build.

Linux and Windows development should go in parallel as easy as possible.

Edit: Please also refer to the official build documentation.

Building CSync

To build mirall, csync must be built first. CSync is hosted in its upstream git repo and there is also my development branch which holds the latest changes.

Overview of Build Directory setup.Clone the csync branch into a directory. In parallel to the cloned csync dir, create a new directory buildcsync as a cmake build dir. Change into buildcsync and call cmake like this:

cmake -DCMAKE_BUILD_TYPE="Debug" ../csync

and watch its output. You probably have to fulfill some dependencies, make sure to install all the needed devel packages. You will need log4c, iniparser, sqlite3 and for the modules libssh, libsmbclient and neon for the ownCloud module. Once cmake succeeds, call make to build it. So far relaxed for Linux.

To build csync for Windows, there are a couple of possibilities. The one I chose was to cross compile with mingw under openSUSE. That way I can build for all on one devel machine under my prefered system.

For that, I installed the cross compile and mingw32 packages from the openSUSE Build Service, which really demonstrates power here. I used the mingw repository. Kudos at this point to Dominik Schmidt, a Tomahawk developer, who helped me a lot to set all up and to all people who work in OBS to maintain the mingw repo.

Basically the cross compiler and libs (eg. packages mingw32-cross-gcc, mingw32-gcc-c++ and mingw32-cross-cpp) and the dependencies for the software to build have to be installed from the mingw repo. An action item is left to dig which in detail.

After installation you should have some mingw32-tools such as mingw32-cmake which should be used to build for win.

Now create a directory win and within that again buildcsync. In there, start cmake with

mingw32-cmake -DCMAKE_BUILD_TYPE="Debug" -DWITH_LOG4C=OFF ../../csync

That should do it. I did not find log4c for Win32, so I disabled it in the cmake call.
Now build it with mingw32-make and see if it creates a dll in the src subdir and csync.exe in the client dir.

Building mirall

For mirall, it works similar. Mirall uses Qt and is C++, so again a lot of packages to install. Make again sure to have the mingw32-qt packages, for example mingw32-libqt4-devel and more.

However, there are two caveats with mirall:

  • the current development state of mirall needs the latest devel version of csync which
    we just built. I tweaked the CMakefile that way that if the mirall- and csync and build*
    folders are in the same directory, the csync is found by mirall cmake in the parallel dir.
    So I do not have to install the devel version of csync in my system.
  • to build mirall for windows, it must be made sure that cmake finds the mingw32 Qt tools like moc. Since there is also the Linux moc in the system, this can confuse.
    Domme pointed me to a script that sets some variables correct values to prevent mixing:

    cat ../docmake.sh
    # %_mingw32_qt4_platform win32-g++-cross

    export QT_BINDIR=/usr/bin
    export BIN_PRE=i686-w64-mingw32

    /usr/bin/mingw32-cmake \
    -DCMAKE_BUILD_TYPE="Debug" \
    -DQMAKESPEC=win32-g++-cross \
    -DQT_MKSPECS_DIR:PATH=/usr/i686-w64-mingw32/sys-root/mingw/share/qt4/mkspecs \
    -DQT_QT_INCLUDE_DIR=/usr/i686-w64-mingw32/sys-root/mingw/include \
    -DQT_PLUGINS_DIR=/usr/i686-w64-mingw32/sys-root/mingw/lib/qt4/plugins \
    -DQT_QMAKE_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-qmake \
    -DQT_MOC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-moc \
    -DQT_RCC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-rcc \
    -DQT_UIC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-uic \
    -DQT_DBUSXML2CPP_EXECUTABLE=${QT_BINDIR}/qdbusxml2cpp \
    -DQT_DBUSCPP2XML_EXECUTABLE=${QT_BINDIR}/qdbuscpp2xml ../../mirall

With that setup I can build both the Linux and Windows version quite easily. There is still a lot to be solved, such as automatted packaging and such. CMake as usual is a great help.

On the road…

March 16, 2012 3 comments

Busy times currently as we’re heading to releases of ownCloud. The syncing client I am working on plays a role for that and thus I was doing a lot of work on csync upstream and mirall to get that going. Meanwhile I succeeded to get mirall running under Win32, it was quite an experience for me digging through the mud of various compilers and environments and the different understandings of C standards, especially since csync is plain C and never run on Win32 before.

Last week I have been on CeBIT for a day, which is a huge and busy event. We had a nice preview running there with syncing between the web client and the desktop clients on Windows and Linux, as well as on an Android device. That already worked quite ok, nevertheless I got quite some bug reports since then. And I still have to work with csync upstream to get patches in there before we’re ready for a public beta.

This weekend I will be at the Chemnitzer Linuxtage together with friends from the openSUSE project. I will show some ownCloud syncing there as well, so if you like show up at the openSUSE booth for a demo and a chat 🙂

Categories: ownCloud Tags: , , , ,

Status ownCloud Desktop Syncing

February 17, 2012 11 comments

It has been a busy week where I was working on the ownCloud Desktop sync client. As described in my last post this will be based on a csync module to handle ownCloud as well as mirall, a Qt based desktop client to sync.

That involved basically three things: A csync module, a patch in ownCloud and work on mirall.

For csync, there needs to be a module that handles the connection to ownCloud, for code see [1]. ownCloud has a WebDAV interface, and its used for this. Csync requires the the implementation of a couple of POSIX functions such as opendir, readdir, closedir, stat and these kind of things. I did that using WebDAV equivalents.

Unfortunately one method is needed for the sync algorithms which has no equivalent in WebDAV, which is utime. utime sets the modification time of a file. csync needs that to fix the time after a file was copied to the other side.  For that, I made a little patch to ownCloud to allow a PROPPATCH method on the modified time of a resource. That makes csync happy and working, but that also means that the module is not a general csync webdav module but a csync owncloud module. Named it accordingly.

The first action I started on mirall this time is a bit cleanup. First, I introduced a class  to move all the code which loads, edits and monitors the folders out of the main application class, which was long overdue. That separates the sync folder management in mirall from the GUI stuff. Furthermore I added a few helper classes such as a config file class for mirall. Ah yes, a theming class was also added to handle the future two faces character of mirall easily, as said before mirall should either appear as traditional mirall client, or as ownCloud client. Duncan and me agreed to build that from one source, the theming class supports that.

Last but not least I polished the wizard in which a user connects to his ownCloud and added a test call that immediately evaluates the url and credentials. For the curious, the code is in my work repository but I promise to move it to the ownCloud repo on gitorious soon.

So far for now, if you have suggestions for me I’d love to hear them.  Thanks 🙂

[1] Klaas’ WIP branch csync

Categories: ownCloud Tags: , , ,