Posts Tagged ‘status’

Sync Progress Display

July 23, 2013 16 comments

here is something new and eye candy in the ownCloud Client, so let me show a bit of what we have worked on recently.

Many users of the ownCloud Client were asking for sync progress information, in fact there was none at all until today which is a bit boring. The reason why we hadn’t it was simply that csync, which is the file synchronizer engine we use, did not have an API to hand over progress information of an actual up- or download to higher levels of the application.

We implemented two callbacks in csync: One that informs about start, end and progress of an up- or download of an individual file. Another one processes the overall progress of the currently running sync run, indicating for example that eight files have to be processed, current is file number four, and x of the overall sum y bytes have been processed already.

That information is passed into a singleton class called ProgressDispatcher in the client code, where other classes can connect to a signal providing that information. That comes handy as we need the information at different places in the client GUI.

Sync Progress Display

Sync Progress Display

The screenshot shows the first and probably most detailed implementation of the progress display in the sync accounts details which are part of a new settings- and status dialog.

The visual appearance was worked out by our interaction designer Jan. We always have to argue a bit because obviously I am the greatest interaction designer around 😉 but well, finally we do like Jan says and the result is great IMO. It’s great that we have him for ownCloud as that guarantees that our project is not drifting too much onto the geeky techy side which (I heard) is scaring off some users…

Hope you like it! You can get a preview in the current nightly builds (win, mac, Linux) of the client! Please report bugs as you find them, thanks.

What’s on in Nightly?

April 29, 2013 6 comments

We ownCloud Client devs were quite busy doing bug investigations and two releases the last couple of weeks. But that does not mean that nothing went into feature branches. It did, and now we can start to think of a release 1.3.0 with a couple of new features for the client.

Here is a small overview of what went into ocsync master (actually the dav branch) and into mirall master.

Note that this is alpha state software, but it’s already available for testing from our nightly builds at the ownCloud nightly repository.

If you enable the nightly package repository on your system, please be sure to use the machine for testing purposes only. Windows and Mac nightly builds can be found here to download.

csync: Big File Chunking

Handling big files over HTTP is difficult, if files grew bigger than a gigabyte, we faced multiple problems. To deal with that, we thought of splitting up big files and transmit the parts to let the server finally assemble the file again once all parts are there.

We wrote a little library to implement that, called httpbf. It is now integrated in the current development branch. You will see files big files split up into parts for transmission to the server. With that there should not be a filesize limit to work with any more, except the limits we have from the operating system etc.

Up- and Download Resume

With big file chunking, we also introduce up- and download resume. If a file transmission was interrupted and is started over again later, only the missing parts are going to be pushed again. However, we’re not yet transmitting only changed chunks for existing files yet, just to answer that question here before it comes up. That will be subject of more enhancements.

Source File Verification

Another problem with big files is that if one gets copied into the synced dir on the local machine, the copy is not finished before the sync acutally starts. That can result in useless transmissions because inconsistent data is synced.

We added code to the httpbf library that checks for the integrity of the source file. When the first chunk is transmitted, it is checked if the file has still the size and mod time from the beginning. If not, the library waits for two seconds and starts over with a check again. That way, it waits until the source file has stabilized. A final check if the source file hasn’t changed during upload is done for every file when the upload is finished.

CSync Context Reuse

There is the very unpleasant problem on some distros (namely Ubuntu and Fedora) that the ownCloud Client wastes file descriptors. We figured that this is because the system libraries on that platforms fail to unload the csync ownCloud module properly.

Version 1.2.5 contains a quick fix for that, but the real fix is to reuse the csync context within mirall and not build it up and down for every sync run as we did until now. So we developed the function csync_commit which commits the results of the sync run to the database and leaves the csync context intact for the next run. That way we just load and unload the ownCloud module on time and not every 30 seconds. The csync patch already went upstream.

Mirall: New Setup Dialog

newsetup_sThere is also the new setup dialog in the nightly builds: It is much more simplified compared to the old one, user only needs to add the url and user credentials and thats it. Also, it syncs the whole ownCloud to a local folder, just like other boxes do. That is one of the most often heard enhancement requests, and here we go.

That all I think would qualify for a good 1.3.0 release, but we need more time to fine tune,polish, test and bugfix. But we would appreciate if people could help test. Please report bugs in the client’s bugtracker.
We need to find bugs prior the release, and that happens with your help!

Have fun & Thanks a lot for testing!

Csync for ownCloud Client 1.1.0 – A New Sync Engine

October 11, 2012 18 comments

Along with todays ownCloud 4.5 release we released the new ownCloud Client 1.1.0 with a new syncing concept.

This blog will shed some light on the details. I apologize, it’s a long read.

Time Issues

ownCloud Client versions 1.0.x worked with csyncs traditional way of using the file modification times to detect updates between the two repositories that should be synced to each other. That works fine and conforms to our idea to ideally not use any other metadata in syncing than what the file system has anyway.

Time flies

However, there is one drawback which we all know from daily life: If at least two parties sync on time its important that all clocks are set exactly the same way. Remember good crime movies where a bank robbery always starts with a clock adjustment of all gangsters? We have exactly the same in ownClouds syncing: All involved have to have the same time setting, otherwise modification times of files can not be compared reliably.

There are solutions for computers to set the exact time (like ntp) so in general that works. However, in real life scenarios these are not reliable because either people do not have them started on the system or because the daemon updates the time once in a while and in that time span the clock skews already too much.

Users all the time reported problems with that and other experts continued to advise that we never get around that problems if we don’t change something fundamental and go away from pure time based syncing.

Well, we did that with our csync version 0.60.0 which is the sync engine for ownCloud Client 1.1.0.

An Unique Id

Now, every file and directory inside a sync directory has an unique Id associated. The idea is that the Id changes if the file changes. So in the sync process the need for a file update in either direction can be computed by comparing the two Ids of the file. If the id has changed on one repository the file was changed there and needs to be synced to the other side.

The Ids are generated on the ownCloud server and one challenge for the client is to always download the correct Id of a file. The Ids are just random tags for a file version. It is not associated to the file content as MD5 sums would be. Actually it was a frequent advise to use MD5 sums or a similar approach which digests the files content to detect updates. That would have come very handy because that means comparing file contents directly and, more important, it’s reproducable on either side. Also the client would have been able to recalculate the MD5-Sum of the local files and would not have depended on a local database with Ids that were pulled from the server before.

But we decided against hashes. Calculating MD5-Sums is costly in terms of CPU and time, especially for large files. The CPU problem is small on clients, but not on servers where a lot of clients connect to. Even though the sums can be calculated during upload, the problems remain for the case where the server does not see the upload stream, think of the “mount my Dropbox” case.

For files on the ownCloud server, the Id is always updated when the file gets updated. On the client side the last Id of a file is in the client database. It is invalidated in case the files modification time changed meanwhile to detect local changes.

Change Propagation

Another remarkable change in the 1.1.0 client is that change events in the file tree propagate up to the top directory on the owncloud server, ie. if a file changes in a directory, the id of the directory changes as well as the one of its parent directory etc.

That means that to detect if a file tree has changed, it’s enough to check the top most directories Id. If that has changed, ok, than the client needs to dig deeper, but in the not so rare case that nothing has changed, the one call is enough to detect that. That dramatically lowers the server load with clients because instead of digging through the whole directory structure what we did with the 1.0.x series
it is a few requests now.

CSync and ownCloud for Success

These are very intrusive changes to csync. For example, we had to add two additional fields to the database, add code that is able to build a representation
of the local file tree from the database and make csync query for the file Ids from
the server if needed. Deep under the hood the updater, reconciler and propagator code needed changes to work with the Ids. All these changes did not go back to csync upstream yet.

To not conflict with the upstream version of csync we decided to rename our csync version to ocsync. But: This is a temporar solution for the time we need to catch up with upstream again. That will take a while until everything is sorted again but we will work on that.

I am are very excited about the new version of csync. But obviously there are other changes in the ownCloud Client 1.1.0 which will be subject of another blog post.

Status ownCloud Desktop Syncing

February 17, 2012 11 comments

It has been a busy week where I was working on the ownCloud Desktop sync client. As described in my last post this will be based on a csync module to handle ownCloud as well as mirall, a Qt based desktop client to sync.

That involved basically three things: A csync module, a patch in ownCloud and work on mirall.

For csync, there needs to be a module that handles the connection to ownCloud, for code see [1]. ownCloud has a WebDAV interface, and its used for this. Csync requires the the implementation of a couple of POSIX functions such as opendir, readdir, closedir, stat and these kind of things. I did that using WebDAV equivalents.

Unfortunately one method is needed for the sync algorithms which has no equivalent in WebDAV, which is utime. utime sets the modification time of a file. csync needs that to fix the time after a file was copied to the other side.  For that, I made a little patch to ownCloud to allow a PROPPATCH method on the modified time of a resource. That makes csync happy and working, but that also means that the module is not a general csync webdav module but a csync owncloud module. Named it accordingly.

The first action I started on mirall this time is a bit cleanup. First, I introduced a class  to move all the code which loads, edits and monitors the folders out of the main application class, which was long overdue. That separates the sync folder management in mirall from the GUI stuff. Furthermore I added a few helper classes such as a config file class for mirall. Ah yes, a theming class was also added to handle the future two faces character of mirall easily, as said before mirall should either appear as traditional mirall client, or as ownCloud client. Duncan and me agreed to build that from one source, the theming class supports that.

Last but not least I polished the wizard in which a user connects to his ownCloud and added a test call that immediately evaluates the url and credentials. For the curious, the code is in my work repository but I promise to move it to the ownCloud repo on gitorious soon.

So far for now, if you have suggestions for me I’d love to hear them.  Thanks 🙂

[1] Klaas’ WIP branch csync

Categories: ownCloud Tags: , , ,