Here is something that might be a little outdated already, but I hope it still adds some interesting thoughts. The rainy Sunday afternoon today finally gives the opportunity to write this little blog.
Recently an ownCloud fork was coming up with a little shiny box with one harddisk, that can be complemented with a Rapsberry Pi and their software, promoting that as your private cloud.
While I like the idea of building a private cloud for everybody (I started to work on ownCloud because of that idea back in the days), I do not think that this example of gear is a good solution for private cloud.
In fact I believe that throwing this kind of implementations on the table is especially unfortunate because if we come up with too many not optimal proposals, we waste the willingness of users to try it. This idea should not target geeks who might be willing to try ideas on and on. The idea of the private cloud needs to target at every computer user who wants to store data safely, but does not want to care about longer than ever necessary. And with them I fear we only have very little chances, if one at all, to introduce them to a private cloud solution before they go back to something that simply works.
Here are some points why I think solutions like the proposed one are not good enough:
That is nothing new: The hardware of the Raspberry Pi was not designed for this kind of usecases. It is simply too weak to drive ownCloud, which is an PHP app plus database server that has some requirements on the servers power. Even with PHP7, which is faster, and the latest revisions of the mini computer, it might look ok in the beginning, but after all the neccessary bells and whistles were added to the installation and data run in, it will turn out that the CPU power is simply not enough. Similar weaknesses are also true for the networking capabilities for example.
A user that finds that out after a couple of weeks after she worked with the system will remain angry and probably go (back) to solutions that we do not fancy.
One Disk Setup
The solution comes as one disk setup: How secure can data be that is on one single hardisk? A seriously engineered solution should at least recommend a way to store the data more securely and/or backup, like on an at homes NAS for example.
That can be done, but requires manual work and might require more network capabilities and CPU power.
Last, but for me the most important point: Having such a box in the private network requires to drill a whole in the firewall, to allow port forwarding. I know, that is nothing unusual for experienced people, and in theory little problem.
But for people who are not so interested, that means they need to click in the interface of their router on a button that they do not understand what it does, and maybe even insert data by following an documentation that they have to believe. (That is not very much different from downloading a script from somewhere letting it do the changes which I would not recommend as well).
Doing mistakes here could potentially have a huge impact for the network behind the router, without that the person who did it even has an understanding for.
Also DynDNS is needed: That is also not a big problem in theory and for geeks, but in practice it is nothing easily done.
With a good solution for private cloud, it should not be necessary to ask for that kind of setups.
Where to go from here?
There should be better ways to solve this problems with ownCloud, and I am sure ownCloud is the right tool to solve that problem. I will share some thought experiments that we were doing some time back to foster discussion on how we can use the Raspberry Pi with ownCloud (because it is a very attractive piece of hardware) and solve the problems.
This will be subject of an upcoming blog here, please stay tuned.
Even though we just had the nice and successful ownCloud Contributor Conference there have quite some ownCloud releases happened recently. I like to draw your attention to this for a moment, because some people seem to fail to see how active the ownCloud community actually is at the moment.
There has been the big enterprise release 9.1 on September 20th, but that of course came along with community releases which are in the focus here.
We had server release 8.0.15, server release 8.1.10, server release 8.2.8 and release 9.0.5. There are maintenance releases for the older major versions, needed to fix bugs on installations that still run on these older versions. We deliver them following this plan.
The latest and greatest server release is release 9.1.1 that has all the hardening that also went into the enterprise releases.
Aside a ton of bugfixes that you find listed in the changelog there have also been interesting changes which drive innovation. To pick just one example: The data fingerprint property. It enables the clients to detect if the server got a backup restored, and saves changes on the clients to conflict files if needed. This is a nice example of solutions which are based on feedback from enterprise customers community running ownCloud, who help with reporting problems and proposing solutions.
Talking about professional usage of ownCloud: Of course also all the server release are available as linux packages for various distributions, for example the ownCloud server 9.1.1 packages. We think that our users should not be forced to deploy from tarballs, which is error prone and not native to Linux, but have the choice to use linux packages through the distributions package management.
There also have been client releases recently: The Android client versions 2.1.1 and 2.1.2 were released with important changes for Android 7 and much more fixes, as well as iOS client versions 3.5.0 and 3.5.1. The desktop client 2.2.4 also got a regular bug fix update (Changelog).
I guess you agree that is a lot of activity shown in the ownCloud project, making sure to get the best ownCloud experience out there for the users, driven by passion for the project and professional usage in focus.
ownCloud is even more hiring. In my last post I wrote that we need PHP developers, a security engineer and a system administrator.
For all positions we got interesting inquiries already. That’s great, but should not hinder you from sending your CV in case you are still interested. We have multiple positions!
But there is even more opportunity: Additionally we are looking for an ownCloud Desktop Client developer. That would be somebody fluid in C++ and Qt who likes to pick up responsibility for our desktop client together with the other guys on the team. Shifted responsibilities have created this space, and it is your chance to
jump into the desktop sync topic which makes ownCloud really unique.
The role includes working with the team to plan and roll out releases, coordinate with the server- and mobile client colleagues, nail out future developments, engage with community hackers and help with difficult support cases. And last but not least there is hacking fun on remarkable nice Qt based C++ code of our desktop client, together with high profile C++ hackers to learn from.
It is an ideal opportunity for a carer type of personality, to whom it is not enough to sit in the basement and only hack, but also to talk to people, organize, and become visible. Having a Qt- and/or KDE background is a great benefit. You would work from where you feel comfortable with as ownCloud is a distributed company.
The ownCloud Client is a very successful part of the ownCloud platform, it has millions of installations out there, and is released under GPL.
If you want to do something that matters, here you are! Send your CV today and do not forget to mention your github account 🙂
What we do – what you will work on
The call is for people who understand the vision of bringing the idea ownCloud to an enterprise ready level: ownCloud is not only running on individual open source enthusiasts hardware, but also on sites with huge amounts of data like CERN or the Sciebo project, and at large companies who want to work with their data in a secure way.
To provide the best solution for all of them we are looking for:
In this role, you make sure that the infrastructure that we use in ownCloud is up and running. That involves troubleshooting and streamlining existing infrastructure, but also designing new services. If you love virtualization of all kinds and have an eye for security, this position is for you. Of course all this does not only happen behind closed doors, but you will be in contact with the open source community around ownCloud.
For security professionals who would like to take on a high profile open source project. As security is one of the core values of ownCloud, we are looking for somebody who constantly monitors the code flowing in for security problems, is able to find glitches in existing code and handle the bug bounty program. That and more is the task of this high profile position.
For engineers with a passion for good software design and a love for writing code without being code monkeys: In this role you iron the server part of our platform, build new features, work on fixing bugs with the support colleagues and bother the architect with new ideas how to make the thing even better. For this you need to urge to get down and dirty with code, feel yourself comfortable in a team of high profile developers who can teach you things and learn from you.
PHP or what?
Yes, ownCloud is written in PHP, and PHP is the most important, but by far not the only language that we use for the ownCloud platform.
Before you turn your back because of PHP, please think twice. There are a lot of good reasons why we are going with PHP, some of them are named in this blog, but there is more: For example PHP7: With PHP 7 (which can be used with ownCloud) the language has caught up with many criticism it faced before and has done a big leap.
And anyway, the language of a system is not the only thing that is important in a developers life. It is rather how many people use, love and recommend the project and the development processes the team lives. And in all that points, ownCloud is already awesome, and will become even more with your help.
Send your resume in to email@example.com so we can get talking!
A couple of weeks ago we released another significant milestone of the ownCloud Client, called version 2.2.0, followed by two small maintenance releases. (download). I’d like to highlight some of the new features and the changes that we have made to improve the user experience:
Overlay icons for the various file managers on our three platforms already exist for quite some time, but it has turned out that the performance was not up to the mark for big sync folders. The reason was mainly that too much communication between the file manager plugin and the client was happening. Once asked about the sync state of a single file, the client had to jump through quite some hoops in order to retrieve the required information. That involved not only database access to the sqlite-based sync journal, but also file system interaction to gather file information. Not a big deal if it’s only a few, but if the user syncs huge amounts, these efforts do sum up.
This becomes especially tricky for the propagation of changes upwards the file tree. Imagine there is a sync error happening in the foo/bar/baz/myfile. What should happen is that a warning icon appears on the icon for foo in the file manager, telling that within this directory, a problem exists. The complexity of the existing implementation was already high and adding this extra functionality would have reduced the reliability of the code lower than it already was.
Jocelyn was keen enough to do a refactoring of the underlying code which we call the SocketApi. Starting from the basic assumption that all files are in sync, and the code has just to care for these files that are new or changed, erroneous or ignored or similar, the amount of data to keep is very much reduced, which makes processing way faster.
On the ownCloud server, there are situation where notifications are created which make the user aware of things that happened.
An example are federated shares:
If somebody shares a folder with you, you previously had to acknowledge it through the web interface. This explicit step is a safety net to avoid people sharing tons of Gigabytes of content, filling up your disk.
With 2.2.x, you can acknowledge the share right from the client, saving you the round trip to the web interface to check for new shares.
Keeping an Eye on Word & Friends
Microsoft Word and other office tools are rather hard to deal with in syncing, because they do very strict file locking of the files that are worked on. So strict that the subsequent sync app is not even allowed to open the file, not even for reading. That would be required to be able to sync the file.
As a result the sync client needs to wait until word unlocks the file, and then continue syncing.
For previous version of the client, this was hard to detect and worked only if other changes happened in the same directory where the file in question resides.
With 2.2.0 we added a special watcher that keeps an eye on the office docs Word and friends are blocking. And once the files are unlocked, the watcher starts a sync run to get the files to the server, or down from the server.
Advances on Desktop Sharing
The sharing has been further integrated and received several UX- and bugfixes. There is more feedback when performing actions so you know when your client is waiting for a response from the server. The client now also respect more data returned from the server if you have apps enabled on the server that for example
limit the expiration date.
Further more we better respect the share permissions granted. This means that if
somebody shared a folder without create permissions with you and you want to reshare
this folder in the client you won’t get the option to share with delete permissions. This avoids errors when sharing and is more in line with how the whole ownCloud platform handles re-sharing. We also adjusted the behavior for federated reshares with the server.
Please note to take full advantage of all improvements you will need to run at least
server version 9.0.
Last night I found time to finally install the first release candidate of Volumio 2, my preferred audio player software. This is more exciting than it sounds, because when I read the blogpost last summer that Volumio is going to be completely rewritten, with replacing the base technologies, I was a bit afraid that this will be one of the last bits that we heard from this project. Too many cool projects died after famous last announcements like that.
But not Volumio.
After quite some development time the project released RC1. While there were a few small bugs in a beta, my feelings about the RC1 are really positive. Volumio2 has a very nice and stylish GUI, a great improvement over Volumio1. Album-art is now nicely integrated in the playback pane and and everything is more shiny, even if the general concept is the same as in Volumio1.
I like it because it is only a music player. Very reduced on that, but also very thought through and focussed to fulfill that job perfectly. I just want to find and play music from my collection, quickly and comfortable and with good sound quality. No movies, series, images. Just sound.
About speed: While the scanning of my not too big music collection on a NAS was a bit of a time consuming task in the past, this feels now much faster (maybe thats only because of a faster network between the Raspberry and the NAS?). Searching, browsing and everything works quite fluid on an Raspberry2. And with the Hifiberry DAC for output, the sound quality is more than ok.
This is an release candidate of the first release of the rewritten project, and the quality is already very good. Nevertheless I found a few things that did not work for me or could be improved. That the volume control is not working is probably because of the Hifiberry DAC driver, I remember there was something, but haven’t investigated yet.
There are some things in the GUI that could be looked at again: For example on the Browse page, there is the very well working search field. After entering the search term and Enter, the search result is displayed as a list of songs to select from. I wished that the songs were additionally grouped by albums, which should also be selectable to be pushed to the play queue.
Also it would be great if the Queue would somehow indicate which entry is currently played. I could not spot that.
But these are only minor findings which can easily be addressed later after enhancement requests were posted 🙂
I think Volumio2 is already a great success, even before it was released! You should not hesitate to try it if you love to listen to music!
Thanks for the hard work Volumio-Team!
This is the third and final part of a little blog series about a new chunking algorithm that we discussed in ownCloud. You might be interested to read the first two parts ownCloud Chunking NG and Announcing an Upload as well.
This part makes a couple of ideas how the new chunking could be useful with a future feature of incremental sync (also called delta sync) in ownCloud.
In preparartion of delta sync the server could provide another new WebDAV route: remote.php/dav/blocks.
For each file, remote.php/dav/blocks/file-id exists as long as the server has valid checksums for blocks of the file which is identified by its unique file id.
A successful reply to remote.php/dav/blocks/file-id returns an JSON formatted data block with byte ranges and the respective checksums (and the checksum type) over the data blocks for the file. The client can use that information to calculate the blocks of data that has changed and thus needs to be uploaded.
If a file was changed on the server and as a result the checksums are not longer valid, access to remote.php/blocks/file-id is returning the 404 "not found" return code. The client needs to be able to handle missing checksum information at any time.
The server gets the checksums of file blocks along the upload of the chunks from the client. There is no obligation of the server to calculate the checksums of data blocks that came in other than through the clients, yet it can if there is capacity.
To implement incremental sync, the following high level processing could be implemented:
- The client downloads the blocklist of the file: GET remote.php/dav/blocks/file-id
- If GET succeeded: Client computes the local blocklist and computes changes
- If GET failed: All blocks of the file have to be uploaded.
- Client sends request MKCOL /uploads/transfer-id as described in an earlier part of the blog.
- For blocks that have changed: PUT data to /uploads/transfer-id/part-no
- For blocks that have NOT changed: COPY /blocks/file-id/block-no /uploads/transfer-id/part-no
- If all blocks are handled by either being uploaded or copied: Client sends MOVE /uploads/transfer-id /path/to/target-file to finalize the upload.
This would be an extension to the previously described upload of complete files. The PROPFIND semantic on /uploads/transfer-id remains valid.
Depending on the amount of not changed blocks, this could be a dramatic cut for the data that have to be uploaded. More information has to be collected to find out how much that is.
Note that this is still in the idea- and to-be-discussed state, and not yet an agreed specification for a new chunking algorithm.
Please, as usual, share your feedback with us!