ownCloud Client May Hackfest
Again we’re meeting in Berlin at Woboq Intl. Headquarters to work on the ownCloud Sync Client again. One of our topics is the still not completely fixed problem with conflict files. There has been lots of troubles about false conflict files the client is generating in that situations were the ETag database is wasted.
We revisited this problem and will come up with a better solution.
The key changes will probably be
- Conflict files will never again be generated on the server. Even if we are in a conflict situation, we will download the file and keep the conflicting version only on the client. This enables us to detect false conflicts.
- The current way we handle a system time difference has to be changed. We wont adjust the file mtimes of files in the file system any more with the time difference between the client and server. That way we do not suffer from floating time differences any more.
For the decision of which version is more recent, we will still consider the time difference. - We will use a very quick request like OPTION to get the servers time setting to the client. That will allow to calculate the time difference between server and client more accurately. It’s needed to decide which file is more recent.
Read this as a note to self, yet we feel very well fitting into Berlin round may 1st 
What’s on in Nightly?
We ownCloud Client devs were quite busy doing bug investigations and two releases the last couple of weeks. But that does not mean that nothing went into feature branches. It did, and now we can start to think of a release 1.3.0 with a couple of new features for the client.
Here is a small overview of what went into ocsync master (actually the dav branch) and into mirall master.
Note that this is alpha state software, but it’s already available for testing from our nightly builds at the ownCloud nightly repository.
If you enable the nightly package repository on your system, please be sure to use the machine for testing purposes only. Windows and Mac nightly builds can be found here to download.
csync: Big File Chunking
Handling big files over HTTP is difficult, if files grew bigger than a gigabyte, we faced multiple problems. To deal with that, we thought of splitting up big files and transmit the parts to let the server finally assemble the file again once all parts are there.
We wrote a little library to implement that, called httpbf. It is now integrated in the current development branch. You will see files big files split up into parts for transmission to the server. With that there should not be a filesize limit to work with any more, except the limits we have from the operating system etc.
Up- and Download Resume
With big file chunking, we also introduce up- and download resume. If a file transmission was interrupted and is started over again later, only the missing parts are going to be pushed again. However, we’re not yet transmitting only changed chunks for existing files yet, just to answer that question here before it comes up. That will be subject of more enhancements.
Source File Verification
Another problem with big files is that if one gets copied into the synced dir on the local machine, the copy is not finished before the sync acutally starts. That can result in useless transmissions because inconsistent data is synced.
We added code to the httpbf library that checks for the integrity of the source file. When the first chunk is transmitted, it is checked if the file has still the size and mod time from the beginning. If not, the library waits for two seconds and starts over with a check again. That way, it waits until the source file has stabilized. A final check if the source file hasn’t changed during upload is done for every file when the upload is finished.
CSync Context Reuse
There is the very unpleasant problem on some distros (namely Ubuntu and Fedora) that the ownCloud Client wastes file descriptors. We figured that this is because the system libraries on that platforms fail to unload the csync ownCloud module properly.
Version 1.2.5 contains a quick fix for that, but the real fix is to reuse the csync context within mirall and not build it up and down for every sync run as we did until now. So we developed the function csync_commit which commits the results of the sync run to the database and leaves the csync context intact for the next run. That way we just load and unload the ownCloud module on time and not every 30 seconds. The csync patch already went upstream.
Mirall: New Setup Dialog
There is also the new setup dialog in the nightly builds: It is much more simplified compared to the old one, user only needs to add the url and user credentials and thats it. Also, it syncs the whole ownCloud to a local folder, just like other boxes do. That is one of the most often heard enhancement requests, and here we go.
That all I think would qualify for a good 1.3.0 release, but we need more time to fine tune,polish, test and bugfix. But we would appreciate if people could help test. Please report bugs in the client’s bugtracker.
We need to find bugs prior the release, and that happens with your help!
Have fun & Thanks a lot for testing!
ownCloud Client Release 1.2.5
Today is release day!
We just release ownCloud Client 1.2.5, hurrah! It’s a small bug fix release, but it will make more people happy users! It fixes three ugly crashes we found with the help of encouraged bug hunters. One of them was an race between our thread which runs csync and the Qt SSL stack we’re obviously using. That caused a crash which happened after quite some run time, or also never… Hard to reproduce and debug, but finally Danimo and his mighty friends were able to nail that.
The new client can be found through it’s download page, here is the Changelog.
If you find issues, please help us to improve ownCloud by commenting or reporting the bugs on our bugtracker. Thanks!
ownCloud Nightly Builds
as you might have heard, we’re providing nightly builds of the parts of ownCloud, which is ownCloud community server as well as the ownCloud client on its supported platforms Linux, Windows and Mac.
The nightly builds of the client for Windows and Mac and the source balls can be downloaded from the nightly directory on our download server. They are named after the last release, followed by a date time stamp, which is simply a concatentation of the year, month and day the nightly was built on.
For linux we maintain a so called nightly repository in the openSUSE Buildservice which builds for various distributions. That can be added to the linux package management system and that way every morning, an update is there for you. The ownCloud server is built only in the nightly repository from this daily source.
What are nightlies for?
To test ownCloud is a very difficult task. ownCloud is a complex system consisting of variuos parts (anybody thought ownCloud is just a “LAMP web app”?) which supports a huge variety of environments. That multiplies to a huge variety that needs to be tested. We had issues with that in the past and as a result, providing easy access to ready-to-run testing versions is one important action we are taking to improve quality as that enables more people to participate on tests.
Of course nightly builds are not released software, nor got they extensive testing. Nightly builds must never be used on production installations, please, really don’t do.
But maybe you have a test server or virtual machine somewhere in spare. So it would be great if you could install a nightly version on that, combine it with a nightly built client, and see if everything runs as you would expect it.
Another great purpose of the nightly builds is to use it for bug verification. In client development we often comment on bugs with sentences like this is fixed now, please verify with the current nightly build. That way bug reporter get a very quick way to verify the fix of their issue and developers can be sure that their fix really is sufficient.
How are nightlies built?
We are doing the nightly builds through an instance of Jenkins, a continous integration server. With a little script we wrote Jenkins feeds the OBS every night, so this is completely automatted. But more on that in a subsequent blog.
ownCloud Meetup in Nürnberg
Next week a part of the ownCloud team will meet in Nürnberg for a creative time and we thought a little meetup would be cool. Join us for a relaxed evening where we will give short talks about current state of ownCloud, new features and the near future.
Let’s meet
Wednesday, February 27th, 6:00 PM at the Nuremberg Coworking Space.
Talks (short, don’t be afraid!):
- ownCloud overview by Holger Dyroff
- The updated ownCloud 5 User Interface by Jan-Christoph Borchardt
- Next steps for the ownCloud Desktop Clients by Daniel Molkentin
The evening closes with a get-together, your chance to meet the team and discuss in a relaxed atmosphere.
Everybody is welcome!
ownCloud Sync Client 1.2.0 final
Yesterday, ownCloud Client 1.2.0 was released. You can get it from here. We worked on this since end of november last year, you might have seen my other blogs about the beta versions we had for this release.
What is interesting about the release from a more technical point of view? Here are a couple of examples.
One of the things which often was complained about was the performance of the client. Performance is a very broad term, so we have to examine the details: Many people felt like its a performance problem that former clients polled the local file system for changes on the MacOSX and Windows platform. That was recommended from other projects which have experience with going through large file collections. QFileSystemWatcher seemed not to be designed for this usecase.
Polling was ok for desktop computers with up-to-date hardware, but on devices running from batteries this was a energy drain. Good that we could fix that with 1.2.0 using native implementation of change detectors on Windows and MacOSX. They detect changes within a file system tree. If a change is detected, a sync run is started. But note, this is only for local file systems. Detection of changes on the server is a different story which has to be told as well.
Another performance problem was within the file upload which is done by a HTTP PUT request. Month ago (pretty after my start at ownCloud) I implemented it using a tmp file in between. That means, the source file was copied to a tmp file, and from that it was processed to the request body. Improvement was needed and we changed the code to directly read from the file descriptor of the opened source file.
Another thing that was improved with 1.2.0 is the error reporting to the user. Former versions of the client sometimes provided error messages which were not really accurately describing the problem. The reason for that was that csync uses errnos (yes, the ones from errno.h) to name errors as csync maps everything to a POSIX file interface. That surely works as long as you’re on a kind of file system. But it’s hard to map HTTP communication problems onto that. So we decided to add our own “custom” errnos and enhance the whole idea to use these to describe problems. That works more accurate now.
The next things on the list are for example a more convenient setup dialog for the client. Also we will get away from the kind of hard coded target file name on the cloud. A better network recognition will also be next as well as better handling of big files. And more…
Thanks a lot to all who helped to get 1.2.0 finished! It is big fun to work in such a great community 
ownCloud Client 1.2.0 beta2
Yesterday the ownCloud Client team released the ownCloud Client 1.2.0 beta 2. It includes a couple of improvements compared to beta 1 which was released before Christmas.
The release of version 1.2.0 is planned for the next week if things go smooth.
New Sync Protocol Dialog
In particular, the the following improvements were added:
- Proxy authentication fixed (Basic auth, NTLM will not yet work)
- The status dialog now provides statistics on the last sync run (via the info button). It will tell in detail which files have been synced, added or deleted.
- Client will go offline while the server in in maintenance mode (feature available with ownCoud master only)
- Improved SSL Certificate acceptance
- All sizes of the new icons are available.
- Support files > 2 GB on all platforms for uploading.
- Fixed some minor memory leaks and again saved some server requests through optimizations.
- Improved error reporting to the user.
- Remove legacy theming support.
- Windows, 32/64 Bit: http://download.owncloud.com/download/testing/owncloud-1.2.0beta2-setup.exe
- Mac OS X ( >= 10.6, 64 bit): http://download.owncloud.com/download/testing/ownCloud-1.2.0beta2.dmg
- Linux: http://software.opensuse.org/download/package?project=isv:ownCloud:testing&package=owncloud-client
- Mirall: http://download.owncloud.com/download/testing/mirall-1.2.0beta2.tar.bz2
- OCSync: http://download.owncloud.com/download/testing/ocsync-0.70.1.tar.bz2
We would appreciate if you give this release a test ride. Note that because it is beta you should make extra sure to have have backups of your data.
If you want to give feedback, please use our mailing list for general discussion and the issue tracker for bug reports. Please read our new guidelines on bug reporting before!
Download Links:
Sources:
Have fun!
