AdamW on Linux and more (Posts about ownCloud)

Looking for new maintainer for Fedora / EPEL ownCloud packages

2015-08-29T14:27:21Z

So I've been maintaining ownCloud for the last little while. Unfortunately I sat down today to try again and update the package to the latest upstream (8.1.1), and somewhere in the second hour of insanely stupid PHP autoloader code, I just snapped. I can't take this crap any more.

I only personally really needed OC for calendar and contact sync anyway, so I've set up Radicale instead: it's written in Python and it doesn't have a ridiculous forest of bundled crap.

Given that there are dozens of other things I could be spending my time on that I'd find more rewarding, I'm just not willing to do any further major updates of the Fedora / EPEL ownCloud package, I'm sorry. I'm willing to keep the current major versions (8.x in everything but EPEL 6, 7.x in EPEL 6) updated until they go EOL, at which point if no-one else is interested, I will orphan the package.

If anyone would like to take on the work of doing the 8.1 upgrade and maintaining the package in future, please do let me know and I'll happily transfer it over. To do a decent job, though, you are going to need to know or be willing to learn quite a lot of intimate and incredibly annoying details about things like PHP class loading and how Composer works. If you don't, for instance, know what it means for unbundling purposes when a PHP library specifies 'classmap' as the autoload mechanism in its composer.json file, and you're not willing to spend your time learning, you probably don't want to own this package. :)

I'm very sorry to folks who are using it, but I really can't deal with the crap any more. If all you need is calendar/contact sync, there are easier ways. Check out Radicale or something like it.

Upstream does of course provide ownCloud packages in an OBS repo. They do not follow Fedora web app packaging policies or unbundling rules, and probably don't work very well with SELinux. Switching from the Fedora/EPEL packages to the OBS ones is likely to require moving various things around and config file editing and stuff. I'm not going to document that, sorry. If anyone else does, though, that'd be great.

Post-F22 plans

2015-06-09T19:21:57Z

Hi, folks! I've been away on vacation for a few weeks, but I'm back now - if you'd been holding off bugging me about something, please commence bugging now. Of course, I have a metric assload of email backlog to dig out from under.

I'll probably have lots more on the list fairly soon, but just thought I'd kick off with a quick list of stuff I'm intending to do in the post-F22 timeframe:

Replace python-wikitcms' homegrown, regex-based mediawiki syntax parsing with mwparserfromhell (which will imply packaging it)
Improve python-wikitcms' handling of Test Days and add a new tool (a relval analog for Test Days) - initial focus on getting useful stats out, but I may set up a similar wiki template-based Test Day page creation system so you can create the initial page for a Test Day with a single command
Write openQA tests for more of the release validation test cases

I should also add better logging and unit tests to python-wikitcms, fedfind and relval, but let's be honest, I'll probably wind up doing shiny exciting things instead...

I'm also going to update the ownCloud packages to 8.0.4 (everything except EPEL 6) and 7.0.6 (EPEL 6) soon.

LinuxFest NorthWest 2015, ownCloud 8 for stable Fedora / EPEL

2015-05-01T18:09:49Z

LinuxFest NorthWest 2015

As I have for many of the last few years, I attended LinuxFest NorthWest this year. It's been fun to watch this conference grow from a couple hundred people to nearly 2,000 while retaining its community-based and casual atmosphere - I'd like to congratulate and thank the organizers and volunteers who work tirelessly on it each year (and a certain few of them for being kind enough to drive me around and entertain me on Sunday evenings!)

The Fedora booth was extra fun this year. As well as the OLPC XO systems we usually have there (which always do a great job of attracting attention), Brian Monroe set up a whole music recording system running out of a Fedora laptop, with a couple of guitars, bass, keyboard, and even a little all-in-one electronic drum...thing. He had multitrack recording via Ardour and guitar effects from Guitarix. This was a great way to show off the capabilities of Fedora Jam, and was very popular all weekend - sometimes it seemed like every third person who came by was ready to crank out a few guitar chords, and we had several bass players and drummers too. I spent a lot of time away from the booth, but even when I was there we had pretty much a full band going quite often.

It was good to meet Brian and also Pete Travis, who does fantastic things for Fedora Docs, as well as Jeff Fitzmaurice. Jeff Sandys was there as usual as well, so we got to catch up over lunch.

I didn't have a talk this year (I proposed one but it didn't make it through the voting process), so I was able to take it nice and easy and just meet up with folks and watch talks. In between all of that I also got myself 3D scanned by Dianne Mueller, who had herself set up in a trailer with a big lazy Susan and a Kinect and software I know nothing at all about which managed to produce a scarily accurate model of me and my terrible posture. She's promised I'll get a tiny plastic bust of myself in the mail sometime soon, though I'm not sure exactly what to do with it...so thanks, Dianne!

Hard to mention everyone else I ran into or met, but of course there was the (thankfully) inimitable Bryan Lunduke and James Mason of openSUSE fame, who took up their traditional spot opposite us and cried all weekend as they watched the huge crowds flock our booth...

There were a lot of really good presentations. I particularly enjoyed Frances Hocutt's A developer's-eye view of API client libraries, which sounds a little dry but was very well presented and full of good notes for API client library producers and consumers. Frances wrangles API client libraries for the Wikimedia Foundation, so it was good to get to thank her for her work on the list of Mediawiki client libraries and the Gold standard set of guidelines for Mediawiki client library designers and all the other things she's done to improve API client libraries for Mediawiki - obviously this has been invaluable to wikitcms development.

It was also great to meet Frank Karlitschek at his Crushing data silos with ownCloud talk. I've been packaging ownCloud and making small upstream contributions for a while now so I've chatted with several of the devs on IRC and GitHub - I didn't even know Frank was going to LFNW, so it was an unexpected bonus to be able to say 'hi' in real life, and inspired me to do some work on OC, of which more later!

Diane's presentation on the latest bleeding-edge bits of the OpenShift stack went partly over my head - my todo list includes items like 'learn what the hell is going on with all this cloudockershifty stuff already' - but she always presents effectively and it was interesting to learn that OpenShift 3.0 is a real production thing built on Project Atomic, which is a bit astonishing since in my head Atomic is still 'this weird experimental thing Colin Walters keeps bugging me about'. These cloud people sure do move fast. Kids these days, I don't know.

Finally it was great to see Seth Schoen present Let's Encrypt. I'd heard a little about this awesome project but it was good to get some details on exactly how it's being implemented and how it will work. Their goal is, pretty simply, to make it possible to get and install a TLS certificate that will be trusted by all clients for any web server by running a single command. They're basically automating domain validation: the server comes up with a set of actions that will demonstrate control of the domain, the script running on your web server box (the 'client' in this transaction) demonstrates the ability to make those changes, the server checks they were done and issues a certificate, the script installs it. None of it is rocket science, but it's so immeasurably superior to doing it all one awkward step at a time with openssl-generated CSRs and janky web interfaces that the only wish I have is for it to be in production already. The real goal is to enable a web where unencrypted traffic simply doesn't happen - make it sufficiently easy to get a trusted certificate that, simply, everyone does it. It was pretty cool that at the end of the talk Seth was mobbed by Red Hat / Fedora folks offering help with integration - I'm guessing you'll be able to use this stuff on RHEL and Fedora servers from day 1.

All that and we had the now-traditional Friday night games night (beer, pizza, M&Ms, and Cards Against Humanity - really all you need on a Friday night!) too. It was a very enjoyable event as always.

ownCloud 8 for Fedora 21, Fedora 20, and EPEL 7

The other big news I have: ownCloud 8.0.3 came out recently, and it seemed like an appropriate time to kick most of the still-maintained stable Fedora and EPEL releases over to it. So there are now Fedora 21, Fedora 20 and EPEL 7 testing updates providing ownCloud 8.0.3 for those releases. Please do read the update notes carefully, and back everything up before trying the update!

EPEL 6 is still on ownCloud 7.0.5 for now; I'd have to bump even more OC dependencies to new versions in order to have ownCloud 8 on EPEL 6. That might still happen, but I decided to get it done for the more up-to-date releases first.

This build also includes some fixes to the packaged nginx configuration from Igor Gnatenko, so I'd like to thank him very much for those! I still haven't got around to testing OC on nginx, but Igor has been running it, so hopefully with these changes it'll now work out of the box.

Fedora 22 Alpha, Ipsilon Test Day, openQA progress, and ownCloud status

2015-03-11T19:23:44Z

Fedora 22 Alpha

The big news this week is definitely that Fedora 22 Alpha is released! We even managed to ship this one on time, though as it's an Alpha, it is of course not bug-free. For me the nicest thing about Fedora 22 is all the improvements in GNOME 3.16, which Alpha has a pre-release of. The new notification system is a huge improvement. I'm fighting some bugs in Evolution's new composer, but mostly it's working out.

Ipsilon Test Day

We also have the next Fedora 22 Test Day coming up tomorrow, Ipsilon Test Day. This is one of those pretty specialized events that requires some interest in and probably knowledge of a specialist area; it also requires a fairly powerful test system with a decent amount of spare disk space.

Ipsilon provides SSO services together with separate identity management systems; it's being implemented into Fedora's identity/authentication system to replace Fedoauth (or this may have already happened, I'm not quite sure on the timelines!). It's one of those things where you go to some website, and you need to log in, and you get redirected through an authentication system that might be shared with lots of other websites - like when you log in to various Fedora sites through the shared Fedora authentication process.

If you're already interested in FreeIPA and might be interested in looking at web application-focused SSO on top of it, you may well be interested in this Test Day. So come join us in #fedora-test-day on Freenode IRC and help out!

openQA automated install testing for Fedora

Personally, I've been working on our openQA-based automated install testing system for the last few days.

Waaaaaaaaait, I hear you say, Fedora has an automated install testing system?

Well, it does now. We are still expecting Taskotron to be our long-term story in this area, but after the awesome Richard Brown demonstrated the feasibility of getting a quick-and-dirty Fedora deployment of openQA running, some of the Red Hat folks working on Fedora QA in our Brno office - Jan Sedlak and Josef Skladanka - threw together an implementation in literally a few days, as a stopgap measure until Taskotron is ready.

The original couple of deployments of the system are behind the Red Hat firewall, just because it works out easier for the devs that way, but I now have my own instance which is running on my own servers and is entirely public; you can look in on all my experiments there. (Please don't download any really huge files from it - all the images I'm testing are available from the public Fedora servers and mirrors, and I have limited bandwidth).

The Fedora tests and the dispatcher and other miscellaneous bits are available, and there's a deployment guide - click on InstallGuide.txt, I'm not giving a direct link because BitBucket doesn't seem to make it possible to do a direct link to the current master revision of an individual file - if you're feeling adventurous and want to set up your own deployment. We are running our instances on openSUSE at the moment, because packaging openQA for Fedora and making any necessary adjustments to its layout would sort of defeat the object of a quick-and-dirty system. And hey, openSUSE is a pretty cool distro too! We're all friends.

The first thing I did was make the system use fedfind for locating, downloading and identifying (in terms of version and payload) images. This had the benefit of making it work for builds other than Rawhide nightlies.

I have my own forks of the tests and tools where I'm keeping topic branches for the work I'm doing. Currently I'm working on the live branches, where I'm implementing testing of live images (and really just uncoupling the system from some expectations it has about only testing one image per build per arch in general). It's mostly done, and I'm hoping to get it merged in soon. After that I'm probably going to work on getting the system to report bugs when it encounters crashes. It's painful sometimes figuring out how to do stuff in perl, but mostly it's been pretty fun working with the system; you can argue that a test system based on screenshot recognition is silly, but really, just about any form of automated testing for interactive interfaces is silly in some way or another, and at least this one's out there and working.

The openQA devs have been very open to questions and suggestions, so thanks to them! I have a couple of trivial commits upstream to fix issues I noticed while running the development packages on my instance.

When you look at Fedora 22 (and later?) release validation pages, results you see from 'coconut' and 'colada' are from the openQA instances - 'coconut' results are from the deployments managed by the main devs, and 'colada' results are from my deployment.

ownCloud updates

Finally, before I dived into openQA I did find a bit of time to work on ownCloud. I got ownCloud 8 into good enough shape (I think!) to go in Rawhide and Fedora 22: it's now in Rawhide, and there's an update pending for Fedora 22.

The other distro which gets a major update is EPEL 6; there's an update pending for 7.0.4 (current stable for EPEL 6 is OC 6).

7.0.4-3 is also pending as an update for the stable Fedora releases and EPEL 7; for those releases it's a minor update which mostly tweaks the Apache HTTP configuration file handling. I came up with a new layout which, as well as making the App Store actually work, should make it easier to 'fire and forget' a typical ownCloud deployment's HTTP configuration; after you do the initial setup, you can run ln -s /etc/httpd/conf.d/owncloud-access.conf.avail /etc/httpd/conf.d/z-owncloud-access.conf to open up access from anywhere, and you won't have to manually check config files on future upgrades, because I'll keep owncloud-access.conf.avail up to date with any directory/path changes that crop up in future releases. If you have an existing ownCloud install and you'd like to 'migrate', I recommend:

Move any customizations you have to the HTTP configuration into an override file, e.g. /etc/httpd/conf.d/z-owncloud-local.conf
Restore the clean packaged copy of /etc/httpd/conf.d/owncloud.conf
Run ln -s /etc/httpd/conf.d/owncloud-access.conf.avail /etc/httpd/conf.d/z-owncloud-access.conf (assuming you want to allow access from any host!)
Never edit any of the packaged files in future (including the .inc and .avail file(s)), always do customizations in override files

Happy Clouding!

ownCloud 8.0.0 preliminary testing for Fedora 21 / 22

2015-02-23T18:39:51Z

Hey folks!

For any ownCloud'ians out there; I worked some more on the OC 8 packages for Fedora today and I think they're in reasonably testable shape now. I just updated my production deployment and it worked first time.

IMPORTANT: back up really, really carefully before testing these out. They should work, but I want to get more testing to be sure. This is what you'll be doing. Please, please make sure beforehand that if the upgrade breaks completely, you have everything you need to recover. Ideally, test these on a dev instance or test machine or something.

Please ensure you have this line in /etc/owncloud/config.php before updating:

 'check_for_working_htaccess' => false,

or else the OC updater will refuse to run. I will probably add a belt-and-braces fix for this later, but it's a good idea to have that set in any case. It has been part of the default config in the package for some time, but very old installations may not have it.

If you have edited /etc/httpd/conf.d/owncloud.conf at all please merge in the changes from the package before attempting to access the server. There are substantial changes in the Apache config files. I have tried to make them easier to work with. If all you need to do is make the OC installation accessible (override the config that makes it only accessible from localhost out of the box), you do not need to modify owncloud.conf at all, all you have to do is:

ln -s /etc/httpd/conf.d/owncloud-access.conf.avail /etc/httpd/conf.d/z-owncloud-access.conf

if you previously enabled global access in any other way I recommend switching to this one, as it will ensure things keep working in future without you needing to do anything, but you can see the directories you need to allow access to in that file if you want to do it any other way. /var/lib/owncloud/apps must be allowed and must be Aliased to /owncloud/appstore-apps (or else installing apps from the 'app store' won't work, and you're really going to need it to, see below). /var/lib/owncloud/assets should be allowed and should be Aliased to /owncloud/assets (though this only matters if you enable the asset-pipeline.enabled setting in config.php).

Please keep owncloud.conf (and all the .inc files in the package) unmodified if at all possible, and use a separate file to make any overrides you want to the configuration. This way when I need to update owncloud.conf in future you will get the changes automatically.

After upgrading you will find that most of your apps have disappeared. Don't panic! ownCloud just decided to move a lot of apps that were previously shipped in the tarball (including even Contacts and Calendar) into the 'app store'. You can go into the 'Apps' screen as an admin user and enable the apps you need. When you enable them, all your existing data and configuration for them should reappear, they will not be blank.

I am considering whether to provide packages for some of the core apps or even add them back to the ownCloud package itself, but for now, installing them from the app store should work fine - as long as you followed the instructions above about making sure the Apache config snippets are up to date.

The packages are available in my side repo for Fedora 21 and 22 (I've symlinked 22 to 'rawhide' as well, as the 22 builds should probably work for Rawhide). Get the repo file and place it in /etc/yum.repos.d to enable the repo.

ownCloud 8 and Fedlet 22(?) updates

2015-02-22T01:54:20Z

Hi there folks!

Between testing and working on my current work-related-ish pet projects fedfind and python-wikitcms/relval, I haven't had much time for Fedlet or the ownCloud package lately. However, I finally managed to squeeze in a few hours to work on them this week.

Fedlet

The Fedlet repository now has a 3.20-pre kernel build that more or less works, for me, in testing. It seems like perhaps it's buggier than 3.18 was, but it at least built and it boots and it has sound and wifi and power status on the V8P. I did throw in a patch which may help with the RPMB bugs. For me it seems slower than 3.18 and touch input is rather worse than it was before, but I haven't had time to look into that in any detail yet. I have built a test Fedora 22-based image which boots, but given the apparent state of 3.20 and the pre-Alpha state of F22 at present I'm not going to release it.

I also set up a public copy of the actual repo from which the Fedlet kernel is built. You can clone that repo and check out the baytrail branch and you have the actual build bits right there. There's one small caveat: I make a few changes right before actually building which are never checked into the repo. They are:

Add the changelog
make config-release
sed -i -e 's,%define debugbuildsenabled 1,%define debugbuildsenabled 0,g' kernel.spec

Step 2 disables debugging in the kernel build (which you really want, on slow Fedlet-ish hardware) and step 3 says not to build a separate kernel-debug package with debugging turned on. Obviously if you actually want kernel debugging to do some...kernel debugging...you can skip 2 and/or 3.

I don't check those changes in because they tend to make merging from upstream (Fedora kernel package) much messier. Aside from that, though, all the substantial stuff is checked in. Mostly the delta is down to patch files now; there's only a couple of changed CONFIG settings any more.

ownCloud

I'm working on getting the ownCloud packages up to the recently-released 8.0. Major OC releases always require quite a bit of gardening downstream. I now have a side repo for OC 8 available; for now there's only a repo for Fedora 22, but you could use it for Rawhide too, and it'll be easy to add at least Fedora 21 soon.

The repo contains OC 8 packages plus Pimple 3.0 (which OC 8 needs) and Doctrine DBAL 2.5.1 with a change that seems to break ownCloud partly reverted.

So far it's at the point where a clean deployment with sqlite as the database works, and you can install apps from the 'app store'. This is important as some apps that were previously part of the core - notably Contacts, Calendar, and Documents - have been moved to the store. I am not sure whether to provide packages for these or even roll them back into the main owncloud package, but for now I'm following upstream.

I haven't yet tested any other database, and I have not tested upgrading an existing install. It goes without saying that you should only do anything at all with this repository on a test setup. :)

You can use oc8-httpd-sqlite.ks to do a full turnkey test install; as usual for my OC test kickstarts, running an F22 install with that kickstart will give you a box with OC running and accessible from any system right out of the box, with insecure admin account (admin/admin) and database credentials, never use my test kickstarts unmodified in production or on any box which is publicly accessible. The kickstarts for other databases need a quick tweak which I'll get around to tomorrow as I test them.

As part of the OC 8 work I wound up overhauling the Apache config layout quite heavily; I needed to fix a few bugs and update it with the latest upstream changes, and took the opportunity to use some Include directives to make it rather cleaner. I've sent a modest proposal about standardizing config snippets intended for inclusion to devel@.

And with that, to bed!

ownCloud 8.x preview packages for Fedora Rawhide

2014-12-30T20:05:10Z

Oh hey there, good lookin' - are you on the hunt for some instability in your life? Well look no further, I've got just the thing for you.

I've got a very experimental ownCloud 8 git snapshot repo up, mainly for me to use in working on the OC 8 packages. At present it's only for Fedora Rawhide (22). There's some hackiness to it - ownCloud doesn't (so far as I can tell) post its build scripts anywhere, and it splits the stuff that's usually shipped in the tarballs across a bunch of git repos, so the source is actually a bunch of git checkouts sloppily piled into one tarball.

The repo has builds for versions of some dependencies I don't want to push to Rawhide at least until official OC 8 pre-releases are out, as well.

As of right now, OC 8 contains some technically non-free code (see previous post for lots more gory details on that). The package currently in the repo still uses the technically non-free JSMin, but it certainly won't go into official Fedora that way; I'm working with upstream to switch to a minifier with a non-problematic license, and they've said they'll take care of it for the 8.0 release.

The package doesn't have a changelog and I didn't bother changing the License: field to reflect the JSMin license.

I've put up an OC8 variant of one of my test deployment kickstarts. If you install from a Rawhide boot.iso with inst.ks=https://www.happyassassin.net/ks/oc/oc8-httpd-sqlite.ks you should wind up with a working (and hilariously insecure, so don't attach it to the public internet!) OC 8 deployment - just browse to http://(host)/owncloud and you'll be at the main interface, logged in as 'admin' with password 'admin' (system root password is '111111' - I told you it was insecure).

I was able to successfully upgrade a bare stock OC 7 + sqlite install to OC 8, but haven't tested with a populated instance or any other databases yet. I really, really, really, really, really heartily recommend you don't let this within fifty yards of production data, production systems, or a publicly accessible host.

The asset pipelining stuff - which concatenates and minifies OC's JS and CSS - isn't actually enabled by default, but if you want to try it out, you can edit /etc/owncloud/config.php and change the asset-pipeline.enabled setting to true. You will also need to configure Apache to allow access to the directory /var/lib/owncloud/assets in the usual way.

Adventures in PHP web asset minimization

2014-12-29T19:45:14Z

When I've had time for 'side projects' lately I've mostly been working on preparing the ground for ownCloud 8 in Fedora, trying to get out ahead of upstream changes and package new library dependencies.

ownCloud's web asset minification

Today I wound up looking at OC's new web asset minification stuff. A few releases back OC delivered CSS and JavaScript itself and minified them in real time, using minifiers ripped out of the Mediawiki source code. That was pretty ugly.

In ownCloud 7 they switched to using Assetic for asset management, which is a lot better. It didn't do any minification at that point, though.

With the PR linked above, OC 8 does minification using Assetic filters. This means it grows some new dependencies on the minimization libraries that back the Assetic filters.

Not surprisingly, this being the world of PHP, there's some fun craziness here. Craziness follows - but if you're interested in a performance comparison of available PHP web asset minifiers, skip it and look down a bit further.

The filters ownCloud currently uses are Assetic's CssMinFilter and JSMinFilter. To back those, ownCloud's composer.json now pulls in mrclay/minify and natxet/CssMin.

natxet/CssMin

I started out by taking a look at natxet/CssMin, which isn't too horrible. It loses some points for not being sanely laid out (there's one source file with a whole bunch of classes in it and it just uses classmap autoloading...), but basically it minifies CSS and that's it. It was fairly simple to build a package. It claims to be just a github mirror of this 'cssmin' project, but in fact it's clearly being actively maintained and has developed on from that project; I'm treating it as its own project, now. It doesn't appear to be a port or rewrite of any other minifier.

mrclay/minify and its exciting assortment of minifiers

Then I looked at the JS minifier, though, and started to encounter the crazy. mrclay/minify looks, to my monkey eyes at least, like a fairly hairy ball of olde-worlde PHP craziness. It's not just a minification library, or even several minification libraries. In its own words, it's "an HTTP content server", i.e. it's sort of trying to do the same thing as Assetic, only it's a rather older and smaller implementation. But because choice is oh so tasty, it includes at least three CSS minifiers and two JS minifiers, and lets you pick whichever you like!

It has its own CSS minifier, Minify_CSS_Compressor.
It includes the github project YUI-CSS-compressor-PHP-port - which is, as the name suggests, a PHP port of the CSS bit of Yahoo's YUI Compressor, which is written in Java. This is named CSSmin.php.
It also, for no goddamn reason I can see, ships a different but far less complete port of the YUI Compressor, as lib/Minify/YUI/CssCompressor.php.

For JavaScript:

It includes JSMin.php. This (so far as I can figure out) originated as a port of Douglas Crockford's JSMin (which is written in C), by Ryan Grove. It lived at Google Code until it was moved for license reasons (see long bit below!) to GitHub, where it is now very definitely unmaintained. mrclay has continued to work on it in his tree since it was abandoned by Mr. Grove, so the mrclay/minify version is now a fork. Again see below for details, but briefly, this code is clearly non-free by Debian and Fedora standards, as it imposes a 'field-of-use' restriction.
It includes JSMinPlus.php. This is a copy of JSMin+, which is by all appearances abandoned by its author (though there were two years between the releases of 1.3 and 1.4, so who knows). Despite the name, JSMin+ is a completely different project from (anything else at all called) JSMin, it is not an unofficial successor nor is it based on JSMin (again, whatever you consider to be 'JSMin' exactly - the original code, or any of its various ports) in any way.
It includes lib/Minify/ClosureCompiler.php, which is a wrapper around the offline Java version of Google's Closure Compiler - you're expected to provide it with a copy of the CC .jar file.
Not at all confusingly it also provides lib/Minify/JS/ClosureCompiler.php, which is an entirely different way to use the Closure Compiler - this time via Google's public REST API for it.

When I say it 'includes' or 'has' something, of course, I don't mean it sensibly expresses a dependency on a copy of it, or anything. In classic fashion for the PHP 'Wild West' interregnum between the reigns of PEAR and Packagist/Composer, copies of the libraries are just dumped willy-nilly into minify's tree, not in any particularly obvious layout (in fact most of them appear to be PSR-0 compliant relative to lib/, but you have to poke about a bit to figure that out), without any kind of manifest to explain clearly what they all are, never mind what they're for.

Interlude: Adam is sad

So I was, obviously, rather reluctant to just package up the mrclay 'minify' for Fedora simply in order to satisfy ownCloud's need for a JavaScript minimizer. This kind of haphazardly bundled source tree is a goddamn nightmare to package in a guideline-compliant way just in general, and the redundancy was too absurd for me to swallow - if you're not keeping count, so far we've found that minify has three CSS minimizers, two JavaScript minimizers, and two different ways you can use yet a third JavaScript minifier it doesn't directly include. And don't forget that natxet\CssMin is not the same thing as minify's CSSmin.php (which, if you've forgotten, is tubalmartin's port of the YUI compressor). If I'd just gone ahead and packaged minify, I'd have been packaging six minifier libraries and eight minifier approaches in order to provide ownCloud with two minifiers. No. Not happening.

My next thought was to just package the actual minifier ownCloud uses, via Assetic's JSMinFilter, which is JSMin.php - not any of the other bits of minify. It looks like it'd pretty much lift right out (not surprisingly, as it was initially a third party lib that minify dumped into its tree, see above). At this point, though, I run into another rather larger problem which I mentioned briefly above.

JSMin, freedom, and the JSON License

JSMin.php is not free. This is because the original, Douglas Crockford-written JSMin is not free; no port or derivative of it is free either. It's licensed under the MIT license, but with an added clause:

"The Software shall be used for Good, not Evil."

There's a page where the story of JSMin being kicked off Google Code for this is related, which includes an excerpt from a talk the author gave where he plays the whole situation for laughs and talks about how ha ha funny it is that Google and IBM want him to allow them to use his software for evil, and that's all nice and cute, but the best legal advice available to Debian, Red Hat and (to my knowledge) all other authorities on software licensing is that such restrictions clearly make the software non-free. License wonks refer to this license as 'the JSON license' and it is explicitly listed as non-free by the FSF, Fedora, and Debian (it's hard to find a good reference for DFSG determinations, that's the best I could get).

So, I can't possibly have Fedora's OC package use that code regardless of how or where it comes from. Darn.

Yet more minifiers (no, of course we're not done yet)

So I started looking into alternatives, which (me being me) turned into a bit of a review of PHP web asset minifiers. Here's the ones I found beyond those listed above:

JSqueeze, a native PHP JS minifier
JShrink, a different native PHP JS minifier
matthiasmullie/minify, a native PHP JS and CSS minifier (not at all the same thing as mrclay/minify)

There may have been others, but I'm sort of losing the will to live at this point. As I was planning to suggest ownCloud switch away from JSMin, it seemed prudent to check out all the alternatives, so I hacked up a script for doing some quick and dirty benchmarking on minifiers. (If you want to try it, you'll have to download all the minifier libs and dump them in the directory with the script, dump some .css and .js files in the same directory, and edit CSSmin.php to rename the class to CSSmin1 so it doesn't conflict with CssMin.php - I told you it was dirty).

Benchmarking minifiers

I dumped all ownCloud's CSS and JS assets (minus translations) into a directory and wrote a thing which just calls each minifier on all the files for its asset class, saves the output with a unique extension, and tells me how long it took. Then I compared the sizes of the minified files. The contenders are:

JS

mrclay/minify's JSMin.php, (today's git master)
mrclay/minify's JSMinPlus.php, which I believe is still identical to upstream JSMin+ 1.4
JSqueeze (today's git master) - tested as 'strong', with its global var, method and property renaming enabled, and 'safe', with it disabled
JShrink (today's git master)
matthiasmullie/minify (today's git master)

CSS

mrclay/minify's CSSmin.php (today's git master), which is tubalmartin/YUI-CSS-compressor-PHP-port
mrclay/minify's Minify/CSS/Compressor.php (today's git master)
natxet/CssMin (today's git master)
matthiasmullie/minify (today's git master)

My CSS results are probably a bit questionable as OC has fairly little CSS, but here they are:

[adamw@adam jstest]$ php ./mintest.php 
CSSmin (tubal) time: 0.15763902664185
CSS Compressor time: 0.099493980407715
natxet time: 0.51680493354797
MM minifiy (CSS) time: 0.090332984924316
[adamw@adam jstest]$ for i in css compressor cssmin natxet mmmcss; do echo $i; du -c *.$i | grep -i total; done
css (original)
360 total
compressor (CSS Compressor)
344 total
cssmin (CSSmin tubal)
340 total
natxet (CssMin natxet)
344 total
mmmcss (MM minify)
344 total

Compressor and Matthias Mullie's minify are the fastest, tubalmartin's CSSmin is a bit slower, and natxet/CssMin is quite a lot slower...but in practice, for OC's purposes, they'd all run pretty much instantly, the 'slowest' takes half a second. They're all almost the same in file size - tubalmartin's 'wins' by 4KB but that's probably in the margin of error (which I don't know how to calculate cos I'm not a scientist), and they only save about 5% on the original size, not really worth crowing about.

The JS results are rather more interesting, as OC has a lot of JS.

JSMin time: 13.471320867538
JSMinPlus time: 15.368160009384
JSqueeze (strong) time: 6.8297760486603
JSqueeze (safe) time: 6.8371610641479
JShrink time: 11.497622013092
MM minifiy (JS) time: 19.345649003983

js
3364 total
jsmin
2468 total
jsminplus
2408 total
jsqueeze.strong
2192 total
jsqueeze.safe
2220 total
jshrink
2468 total
mmmjs
2464 total

I noticed that JSqueeze's global renaming feature seems to cause quite a lot of issues - half the bug reports in its github repo seem to be related to it - so it seemed prudent to test it both ways: the 'safe' version is simply $jsqueeze->squeeze(file_get_contents($filename), $specialVarRx=false) instead of $jsqueeze->squeeze(file_get_contents($filename)) (the 'strong' version).

JSqueeze rather beats the pants off of everything else, there - it's nearly twice as fast as the closest competitor, and does substantially better on file size, even in its 'safe' version.

edit 2014-12-30: After initially writing this post I managed to get a test instance of ownCloud git master up and running and tried plunking JSqueeze in in place of JSMin and, lo and behold...it had a bug.

edit 2015-01-01: p@tchwork fixed the bug mentioned in the previous edit; JSqueeze 2.0.1 works well with ownCloud in my testing. 2.0.1 also namespaces the class (which makes the layout a PSR-4 one), and disables the global renaming stuff by default (i.e. 'safe' not 'strong' is now the default). Also, Matthias Mullie substantially sped up his minifier: after that commit his is the very fastest for me, at 5.8328921794891 seconds. Its compression ratio still ranks in the group at the back of the pack, though.

What did we learn on the show tonight, Adam?

PHP developers, for the love of all that is freaking holy, can you all please just goddamn well sit down together and decide on one implementation of trivial things like asset minifiers, and just work on that? Yeeeeeeeeeeeeeeesh, people. And please don't dump your code's third party dependencies into its tree by hand, without an apparent plan or a manifest. At least use Composer, that gives us a fighting chance at figuring out what the hell you've got in there and splitting it out. And, you know, sit down and really think about whether your minifier blob needs seven different goddamned minifier implementations in different places, half of which are variants of each other...

All things considered for JS I'd recommend using JSqueeze, JShrink or Matthias' 'minify', in about that order - those three all seem to be actively maintained and following good current practices. Avoid JSMin due to the licensing issue and because it's not very actively maintained, and JSMinPlus as it doesn't seem to be maintained at all.

wikitcms and ownCloud stuff

2014-12-20T16:48:32Z

For my salary this week I mostly worked on relval/wikitcms - I shipped 1.7 yesterday which is a major rewrite, on top of recent updates which added support for nightly build testing. relval nightly can create the validation pages for nightly composes, relval report-results can report results in them, and relval testcase-stats and relval user-stats can handle them in statistics generation.

Of course there was some work on the magic wiki templates to go along with that, including some little finicky details like getting valid download URLs for the boot.iso images for Rawhide and Branched nightly composes. It all seems to work pretty well, and we've 'nominated' two Fedora 22 nightly composes for testing so far. The nightly currently being tested is 2014-12-16 - here's the results summary page, and I'm hosting hourly-updated testcase-stats info.

(holy crap, I've got the Maple Leafs / Flyers game going on TV while I write this post - I've missed six goals since I started writing it and it's only half way through the first period...)

Today I spent a bit of time working on my packages, mainly ownCloud. Well, I updated roundcubemail to 1.0.4 and sent out updates for that, too. For ownCloud I backported the upstream updates to its Google Drive support (which, er, I wrote...) which make it compatible with v1.x of Google's API library, updated the client library package to 1.0.6, and dropped the ownCloud package's bundled copy of the 0.6 version of the library, which I've been working on forever - good to finally get it done.

Of course, in the mean time the library went up to 1.1.2 and added an autoloader - unfortunately, not in a way that's very consumable for downstreams. So I sent a pull request to improve that, and I've done a scratch build of 1.1.2 with that patch and verified ownCloud works fine with it. I sent a PR for ownCloud to update its bundled copy to 1.1.2.

If anyone who's using this stuff can test the updates:

that'd be a great help.

ownCloud news (7.0.3 coming)

2014-11-10T21:22:37Z

I should have some good Fedlet news soon (later today or tomorrow), but in the meantime, I also spent some time working on ownCloud today.

ownCloud 7.0.3 has been tagged and tarballed upstream, and I've bumped the Fedora and EPEL 7 builds in Koji already. I have it running on my 'test' (that is, er, production) server and it looks fine, so I'll probably edit 7.0.3 into the EPEL 7 update and submit new updates for F20 and F21 tomorrow. Rawhide is already on 7.0.3.

I also sent a chunk of work upstream. I've been trying to get a pull request merged which updates ownCloud's bundled copy of Google's PHP library for their APIs to the current upstream 1.0 series - they're using an old, unsupported and kinda buggy release. I really just wanted to do this to make the library easier to unbundle for Fedora, but somehow it's turned into a bit of work involving fixing some bugs in the ownCloud code that uses it - OC can use Google Drive as an 'external storage' source, i.e. you can map a Google Drive account to a directory in an OC install - and dealing with some issues in the PHP library itself.

So aside from the big(gish) PR which updates the library I submitted a few other misc fixes based on unit testing which should actually make the thing work a lot better than it did before. Not that I'm actually at all interested in Google Drive, but hey! I'm hoping this will provide some motivation to get the 1.x update PR merged and then I can drop one more bundled dep from the OC packages.