Archive

Posts Tagged ‘mirall’

ownCloud Client 1.7.0 Released

November 8, 2014 30 comments

Yesterday we released ownCloud Client 1.7.0. It is available via ownCloud’s website. This client release marks the next big step in open source file synchronization technology and I am very happy that it is out now.

The new release brings two lighthouse features which I’ll briefly describe here.

Overlay icons

For the first time, this release has a feature that lives kind of outside the ownCloud desktop client program. That nicely shows that syncing is not only a functionality living in one single app, but a deeply integrated system add-on that affects various levels of desktop computing.

Overlay Icons on MacHere we’re talking about overlay icons which are displayed in the popular file managers on the supported desktop platforms. The overlay icons are little additional icons that stick on top of the normal file icons in the file manager, like the little green circles with the checkmark on the screenshot.

The overlays visualize the sync state of each file or directory: The most usual case that a file is in sync between server and client is shown as a green checkmark, all good, that is what you expect. Files in the process of syncing are marked with a blue spinning icon. Files which are excluded from syncing show a yellow exclamation mark icon. And errors are marked by a red sign.

What comes along simple and informative for the user requires quite some magic behind the curtain. I promise to write more about that in another blog post soon.

Selective Sync

Another new thing in 1.7.0 is the selective sync.

In ownCloud client it was always possible to have more than one sync connection. Using that, users do not have to sync their entire server data to one local directory as with many other sync solutions. A more fine granular approach is possible here with ownCloud.

Selective SyncFor example, mp3’s from the Music dir on the ownCloud go to the media directory locally. Digital images which are downloaded from the camera to the “photos” dir on the laptop are synced through a second sync connection to the server photo directory. All the other stuff that appears to be on the server is not automatically synced to the laptop which keeps it organized and the laptop harddisk relaxed.

While this is of course still possible we added another level of organization to the syncing. Within existing sync connections now certain directories can be excluded and their data is not synced to the client device. This way big amounts of data can be easier organized depending on the demands of the target device.

To set this up, check out for the button Choose what to Sync on the Account page. It opens the little dialog to deselect directories from the server tree. Note that if you deselect a directory, it is removed locally, but not on the server.

What else?

There is way more we put into this release: A huge amount of bug fixes and detail improvements went in. Fixes for all parts of the application: Performance (such as database access improvements), GUI (such as detail improvements for the progress display), around the overall processing (like how network timeouts are handled) and the distribution of the applications (MacOSX installer and icons), just to name a few examples. Also a lot of effort went into the sync core where many nifty edge cases were analyzed and better handled.

Between version 1.6.2 and the 1.7.0 release more than 850 commits from 15 different authors were pushed into the git repository (1.6.3 and 1.6.4 were continued in the 1.6 branch which commits are also in the 1.7 branch). A big part of these are bug fixes.

Who is it?

Who does all this? Well, there are a couple of brave coders funded by the ownCloud company working on the client. And we do our share, but not everything. Also coding is only one thing. If you for example take some time and read around in the client github repo it becomes clear that there are so many people around who contribute: Reporting bugs, testing again and again, answering silly looking questions, proposing and discussing improvements and all that (yes, and finally coding too). That is really a huge block, honestly.

Even if it sometimes becomes a bit heated, because we can not do everything fast enough, that still is motivating. Because what does that mean? People care! For the idea, for the project, for the stuff we do. How cool is that? Thank you!

Have fun with 1.7.0!

After the 1.4.0 ownCloud Client Release

September 11, 2013 10 comments

You might have heard, ownCloud Client 1.4.0 was released last week. It is available from our sync clients page for all major desktop platforms, investigate the Changelog.

Danimos Visual Guide has outlined the new stuff in the release already, so no need to repeat it here. You should install and try it, that seems to be the opinion of many people who tried it.

Also people who shared their critical view on the client very publically in the past are much more pleased now with 1.4.0. One example is a recent blog post on BITBlokes. It is a blog about all kind of topics around FOSS. I regularly read it and often share its opinions. He concludes very positively about the 1.4.0 client.

It is good to see the positive feedback overall. That shows a couple of things from my engineering point of view: The concentrated work we continously do on all parts of ownCloud pays off. That is obvious of course, but still nice to see. And our (also obvious) actions to improve code quality such as the consequent use of continous integration, code reviews and such helps to improve quality.

“People are always excited if releases come with GUI changes!” I heard people saying. Well, maybe, but that’s not the whole truth. It also proves for me again is how important UI design and UX is. Me as a knee-deep-developer have an interesting relationship to all UX topics: I always have an opinion. Often a strong opinion. But the results coming out of that have not always been the, well, the most optimal. Very fortunate on the client we work together with our UX guy Jan and the positive feedback also shows how good that is for the software.

But enough of release pride. There is more work to do: The bug tracker is still not empty, the list of feature ideas is long. We will continue to focus on correctness, stability and robustness of syncing, performance and useful features and work on a version 1.5 for you.

These are a couple of concrete points we’re focussing on for 1.5:

  1. we already merged the client code on the new upstream sync version in git.
  2. performace improvements through further reduction of the number of requests and more efficiency in database operations on the client.
  3. we are working on a new propagator component that allows us to do the changes mentioned in 2 more easily.
  4. File manager integration, which means havingn icons in Explorer, Dolphin and friends.

A more detailed list can be found at github.

Thank you for all your help and support. It’s big fun!

Csync for ownCloud Client 1.1.0 – A New Sync Engine

October 11, 2012 18 comments

Along with todays ownCloud 4.5 release we released the new ownCloud Client 1.1.0 with a new syncing concept.

This blog will shed some light on the details. I apologize, it’s a long read.

Time Issues

ownCloud Client versions 1.0.x worked with csyncs traditional way of using the file modification times to detect updates between the two repositories that should be synced to each other. That works fine and conforms to our idea to ideally not use any other metadata in syncing than what the file system has anyway.

Time flies

However, there is one drawback which we all know from daily life: If at least two parties sync on time its important that all clocks are set exactly the same way. Remember good crime movies where a bank robbery always starts with a clock adjustment of all gangsters? We have exactly the same in ownClouds syncing: All involved have to have the same time setting, otherwise modification times of files can not be compared reliably.

There are solutions for computers to set the exact time (like ntp) so in general that works. However, in real life scenarios these are not reliable because either people do not have them started on the system or because the daemon updates the time once in a while and in that time span the clock skews already too much.

Users all the time reported problems with that and other experts continued to advise that we never get around that problems if we don’t change something fundamental and go away from pure time based syncing.

Well, we did that with our csync version 0.60.0 which is the sync engine for ownCloud Client 1.1.0.

An Unique Id

Now, every file and directory inside a sync directory has an unique Id associated. The idea is that the Id changes if the file changes. So in the sync process the need for a file update in either direction can be computed by comparing the two Ids of the file. If the id has changed on one repository the file was changed there and needs to be synced to the other side.

The Ids are generated on the ownCloud server and one challenge for the client is to always download the correct Id of a file. The Ids are just random tags for a file version. It is not associated to the file content as MD5 sums would be. Actually it was a frequent advise to use MD5 sums or a similar approach which digests the files content to detect updates. That would have come very handy because that means comparing file contents directly and, more important, it’s reproducable on either side. Also the client would have been able to recalculate the MD5-Sum of the local files and would not have depended on a local database with Ids that were pulled from the server before.

But we decided against hashes. Calculating MD5-Sums is costly in terms of CPU and time, especially for large files. The CPU problem is small on clients, but not on servers where a lot of clients connect to. Even though the sums can be calculated during upload, the problems remain for the case where the server does not see the upload stream, think of the “mount my Dropbox” case.

For files on the ownCloud server, the Id is always updated when the file gets updated. On the client side the last Id of a file is in the client database. It is invalidated in case the files modification time changed meanwhile to detect local changes.

Change Propagation

Another remarkable change in the 1.1.0 client is that change events in the file tree propagate up to the top directory on the owncloud server, ie. if a file changes in a directory, the id of the directory changes as well as the one of its parent directory etc.

That means that to detect if a file tree has changed, it’s enough to check the top most directories Id. If that has changed, ok, than the client needs to dig deeper, but in the not so rare case that nothing has changed, the one call is enough to detect that. That dramatically lowers the server load with clients because instead of digging through the whole directory structure what we did with the 1.0.x series
it is a few requests now.

CSync and ownCloud for Success

These are very intrusive changes to csync. For example, we had to add two additional fields to the database, add code that is able to build a representation
of the local file tree from the database and make csync query for the file Ids from
the server if needed. Deep under the hood the updater, reconciler and propagator code needed changes to work with the Ids. All these changes did not go back to csync upstream yet.

To not conflict with the upstream version of csync we decided to rename our csync version to ocsync. But: This is a temporar solution for the time we need to catch up with upstream again. That will take a while until everything is sorted again but we will work on that.

I am are very excited about the new version of csync. But obviously there are other changes in the ownCloud Client 1.1.0 which will be subject of another blog post.

ownCloud Client 1.0.1

April 26, 2012 11 comments

This week we prepared another ownCloud sync client (oCC) release for you. oCC is released together with csync 0.50.5, the underlying library on which shoulders oCC stands. Find packages for your distribution.

ownCloud LogoOnly a couple of weeks after the first release we could come up with a quite long changelog, larger than for a maintainance release.

First and foremost, oCC now supports SSL connections and does not store passwords any more, well, in clear text in a config file. Still there is no strong encryption for the password in (as that’s a non trivial if not impossible thing) but its not clear text any more, so accidents should not longer happen. For those who really care and put security over convenience there now is the option to not store the password at all but provide it at startup. The whole password storage will get more attention soon, I did some work on that already, also using kwallet, more on that in another post.

The whole start process of oCC, which is more complex as it might seem as libcsync expects a nice environment with a verified connection to the ownCloud, was cleaned and matured. I had to learn about QNAM but now we seem to be friends.

Efforts also went into the following areas:

  • Error handling: People got confused about error messages from oCC which were not accurate here and than. I added more fine granular error reporting to libcsync so that oCC now can give even more helpful error messages.
  • MacOSX: On the sprint weekend in Stuttgart, we made good progress on that platform, will also soon get a release. Domme got the most out of cmake to get dmg image building going, thanks, you rock!
  • Qt 4.6: As usual one has to learn that current distributions ship not so current Qt versions. Thanks to Thomas, he put quite some effort in porting back to Qt 4.6 – so that Debians friends can also use oCC now.
  • Packages: Check out the community repo on OBS: We now can provide packages for way more platforms including Ubuntu and Debian. Still your help would be appreciated in maintaining.

Speaking about the recent sprint in Stuttgart: It was great to be there, ownCloud has an open, friendly and welcoming community in which you really feel the spirit of working on something new and interesting with cool potential. I really enjoy being here,. Thanks guys for all your work and feedback 🙂

Categories: ownCloud Tags: , , ,

CSync and Mirall Development Setup

March 22, 2012 51 comments

A KDE bagger

Build it!


people were asking how to set up a development setup for the syncing client we are working on to sync local files to ownCloud and vice versa currently, work title mirall. While a website about it is not yet finished, I try to summarize it here. There are some hacks here and there but that’s how I do it today. It will improve over time. Note that this is about a real development setup, not for a production build.

Linux and Windows development should go in parallel as easy as possible.

Edit: Please also refer to the official build documentation.

Building CSync

To build mirall, csync must be built first. CSync is hosted in its upstream git repo and there is also my development branch which holds the latest changes.

Overview of Build Directory setup.Clone the csync branch into a directory. In parallel to the cloned csync dir, create a new directory buildcsync as a cmake build dir. Change into buildcsync and call cmake like this:

cmake -DCMAKE_BUILD_TYPE="Debug" ../csync

and watch its output. You probably have to fulfill some dependencies, make sure to install all the needed devel packages. You will need log4c, iniparser, sqlite3 and for the modules libssh, libsmbclient and neon for the ownCloud module. Once cmake succeeds, call make to build it. So far relaxed for Linux.

To build csync for Windows, there are a couple of possibilities. The one I chose was to cross compile with mingw under openSUSE. That way I can build for all on one devel machine under my prefered system.

For that, I installed the cross compile and mingw32 packages from the openSUSE Build Service, which really demonstrates power here. I used the mingw repository. Kudos at this point to Dominik Schmidt, a Tomahawk developer, who helped me a lot to set all up and to all people who work in OBS to maintain the mingw repo.

Basically the cross compiler and libs (eg. packages mingw32-cross-gcc, mingw32-gcc-c++ and mingw32-cross-cpp) and the dependencies for the software to build have to be installed from the mingw repo. An action item is left to dig which in detail.

After installation you should have some mingw32-tools such as mingw32-cmake which should be used to build for win.

Now create a directory win and within that again buildcsync. In there, start cmake with

mingw32-cmake -DCMAKE_BUILD_TYPE="Debug" -DWITH_LOG4C=OFF ../../csync

That should do it. I did not find log4c for Win32, so I disabled it in the cmake call.
Now build it with mingw32-make and see if it creates a dll in the src subdir and csync.exe in the client dir.

Building mirall

For mirall, it works similar. Mirall uses Qt and is C++, so again a lot of packages to install. Make again sure to have the mingw32-qt packages, for example mingw32-libqt4-devel and more.

However, there are two caveats with mirall:

  • the current development state of mirall needs the latest devel version of csync which
    we just built. I tweaked the CMakefile that way that if the mirall- and csync and build*
    folders are in the same directory, the csync is found by mirall cmake in the parallel dir.
    So I do not have to install the devel version of csync in my system.
  • to build mirall for windows, it must be made sure that cmake finds the mingw32 Qt tools like moc. Since there is also the Linux moc in the system, this can confuse.
    Domme pointed me to a script that sets some variables correct values to prevent mixing:

    cat ../docmake.sh
    # %_mingw32_qt4_platform win32-g++-cross

    export QT_BINDIR=/usr/bin
    export BIN_PRE=i686-w64-mingw32

    /usr/bin/mingw32-cmake \
    -DCMAKE_BUILD_TYPE="Debug" \
    -DQMAKESPEC=win32-g++-cross \
    -DQT_MKSPECS_DIR:PATH=/usr/i686-w64-mingw32/sys-root/mingw/share/qt4/mkspecs \
    -DQT_QT_INCLUDE_DIR=/usr/i686-w64-mingw32/sys-root/mingw/include \
    -DQT_PLUGINS_DIR=/usr/i686-w64-mingw32/sys-root/mingw/lib/qt4/plugins \
    -DQT_QMAKE_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-qmake \
    -DQT_MOC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-moc \
    -DQT_RCC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-rcc \
    -DQT_UIC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-uic \
    -DQT_DBUSXML2CPP_EXECUTABLE=${QT_BINDIR}/qdbusxml2cpp \
    -DQT_DBUSCPP2XML_EXECUTABLE=${QT_BINDIR}/qdbuscpp2xml ../../mirall

With that setup I can build both the Linux and Windows version quite easily. There is still a lot to be solved, such as automatted packaging and such. CMake as usual is a great help.

Status ownCloud Desktop Syncing

February 17, 2012 11 comments

It has been a busy week where I was working on the ownCloud Desktop sync client. As described in my last post this will be based on a csync module to handle ownCloud as well as mirall, a Qt based desktop client to sync.

That involved basically three things: A csync module, a patch in ownCloud and work on mirall.

For csync, there needs to be a module that handles the connection to ownCloud, for code see [1]. ownCloud has a WebDAV interface, and its used for this. Csync requires the the implementation of a couple of POSIX functions such as opendir, readdir, closedir, stat and these kind of things. I did that using WebDAV equivalents.

Unfortunately one method is needed for the sync algorithms which has no equivalent in WebDAV, which is utime. utime sets the modification time of a file. csync needs that to fix the time after a file was copied to the other side.  For that, I made a little patch to ownCloud to allow a PROPPATCH method on the modified time of a resource. That makes csync happy and working, but that also means that the module is not a general csync webdav module but a csync owncloud module. Named it accordingly.

The first action I started on mirall this time is a bit cleanup. First, I introduced a class  to move all the code which loads, edits and monitors the folders out of the main application class, which was long overdue. That separates the sync folder management in mirall from the GUI stuff. Furthermore I added a few helper classes such as a config file class for mirall. Ah yes, a theming class was also added to handle the future two faces character of mirall easily, as said before mirall should either appear as traditional mirall client, or as ownCloud client. Duncan and me agreed to build that from one source, the theming class supports that.

Last but not least I polished the wizard in which a user connects to his ownCloud and added a test call that immediately evaluates the url and credentials. For the curious, the code is in my work repository but I promise to move it to the ownCloud repo on gitorious soon.

So far for now, if you have suggestions for me I’d love to hear them.  Thanks 🙂

[1] Klaas’ WIP branch csync

Categories: ownCloud Tags: , , ,