OpenPanel Development Blog

I’m a strong believer in systems that are easily discoverable, where the structure of the files that make up the system communicate something about the application’s internal structure to the outside world. I tried to do much of the same thing with OpenPanel, but at the end of the day there’s still a lot to go around, so perhaps now’s a good time to write up a bit about the thinking and architecture behind OpenPanel and its components.

When we started the design, we wanted to focus on the following demands:

1. OpenPanel should be modular, and modules should not be restricted to any specific language
2. There should be elementary protection against security defects in module code
3. There should be a possibility of multiple user interfaces (at the very least both a GUI and a command line tool)
4. The system should anticipate the possibility of remote control and clustered integration
5. Configuration provided by OpenPanel should follow the standard configuration practices for the programs involved — no funky business relying on access to a MySQL database
6. It should be possible to dedicate an installation to a single purpose (like mail) without requiring unrelated services (Apache, MySQL) through dependencies

What we came up with, is a design that could best be explained as a domain-specific object database that reflects changes into the configuration of outside programs. The opencore daemon implements this abstraction. At the bottom it has an sqlite database that stores object and class data. Modules are implemented as directories inside /var/opencore/modules that define object classes for the database and the program that executes the necessary changes to outside configuration related to objects of those classes. A JSON-based RPC interface accepts requests through HTTP. This backend port is also used to serve static files related to the web-based GUI.

Obviously, accepting remote RPC requests is something you don’t want to combine with root privileges. Also, since we wanted to keep a low number of privileges exposed directly to module code, running the back-end programs/scripts for module actions was definitely not something we felt should be done as the root user. This is where the authd daemon comes in. It accept requests on a unix domain socket that is only accessible to the opencore daemon. Whenever opencore runs a module program, it will fork to a new process, open a socket to authd, tell authd which module will be executed and drop the privileges needed to open any new connections to authd. The unprivileged module can talk to authd on fd 3, but will be restricted to those privileged actions defined in its module.xml file.

Documentation for the Module API and the RPC Interface is on our site, but some of it may still be a bit rough around the edges.

Bookmark It

The problem with developing front-end applications for web browsers from a back-end developer’s point of view.

In our search for quick (and equally dirty) Javascript solutions for problems like form creation and control, we’ve run across a lot of frameworks. “Let’s use the framework which makes our lives easiest” was the chant, to implement anything third party loosely and, if it breaks, rip it out and either find something better or write something from scratch.

With that in mind we decided to go with ExtJS since it offered a lot of very useful widgets, the widest spectrum of features, decent API documentation and a big user base. Loose implementation of ExtJS’ FormPanel was easy: Create a FormPanel object, render it to an HTML element and forget it ever happened. The wow factor was high at first: All this without any hassle whatsoever, great. Now we didn’t have to worry about this anymore.

Apart from a few minor bugs and glitches, the forms worked fine. That is, until we ran some tests on Internet Explorer. The results kind of shocked us: ExtJS components inside a DIV appeared to have absolute positioning, ignoring the DIV’s overflow property. Also, long loading times and general slowness poured rain on our form parade.

Woes

Apparently there was some dodgy stuff going on inside the ExtJS framework, something intended to make sure it all looks and acts the same in all the major browsers. The long loading times are caused by the copious amount of (needed) ExtJS script infrastructure. Of course it’s still possible to minimize and strip a lot of unwanted script code, but since we were still in the process of developing an application we were not going to let ourselves be slowed down by something which could be postponed until a later stage.

The general slowness — especially in IE and Firefox — was caused by something completely different: If you take a look at the ExtJS API documentation you will find a lot of very elaborate objects. There’s a method for almost anything you want and if something’s missing, chances are you can access it through one of the event handlers. Great stuff. But since you’ll never need all of those nice features (and God forbid you will) you could do without most of it.

What we actually wanted was a form widget framework that performs the dirty footwork for us. We wanted it to be fast, flexible, skinnable and cross-browser. That’s what ExtJS promises. It’s there, it does work, but it’s slow and prone to failure. Since ExtJS is one of the most extensive frameworks around I think it’s rather safe to assume that any other similar widget frameworks will have comparable issues; Blaming this all on ExtJS is just too easy, but it’s safe to say that layering frameworks like ExtJS on top of the Javascript runtime will lead to a product that is rigid, huge and slow.

Barking up the right tree

So what’s really going on here? Why is ExtJS so rigid? What causes this slowness and bloat? Perhaps I’m just not competent enough to write an application for a web browser? Let’s go over the points:

• Rigid — lack of standards : Go quickly from A to B.
• Slow — weakness of Javascript : Go quickly from A to B.
• Huge — demands of industry : Go quickly from A to B.

Our positioning bug wasn’t actually caused by ExtJS proper, but rather by the fact that ExtJS needs to hack around a lot of browser-specific issues. We can go nuts over the fact that this problem exists in the first place, but hey, the fine people at ExtJS did their best to make it work anyway, so no, I don’t think I can point the bullshit ray exclusively at the ExtJS framework. The problem here lies in the lack of standards, caused by browser manufacturers being unable (or even unwilling) to cooperate with eachother.

The complexity of ExtJS objects puts a price on the general performance of the application: For every object there are tons of handlers, methods, properties and configuration options. Javascript was never meant to be used as an OO work horse for big applications, neither in architecture nor in performance.

The reason why the ExtJS framework is so big is that it has to serve many masters with many demands: There should be something for everyone and there are lots of people screaming for an easier way to write applications in Javascript. Is it way too bloated for what a browser can handle? I think it is. But who can blame them? I sure can’t. Using it saved us a lot of time and the demand for a framework like that is apparently big enough to justify its existence. So no, they’re trying their best to make something very decent and respect and credit where respect and credit are due.

Maybe I should just blame myself for being a lazy developer. I don’t really want to know why browser #1 does it like this and browser #2 does it like that. And to make it even worse: I don’t want to know how to fix or circumvent it. I think I’m a lot better off writing applications rather than fixing these kinds of inconsistencies. Does that make me lazy? One could argue so, but come on, imagine writing stuff in C and taking four different compilers into consideration. I think there are a lot of differences between web developers and application developers. For one, web developers usually aren’t anal retentive when it comes down to writing structured code. Application developers generally don’t know jack about CSS (and like it that way).

Conclusion

The lack of standards. IE does it this way, other browsers do it that way. If there would be proper standards life would be a lot easier. HTML and CSS would be more respected by both web developers and application developers. Mastering the skills needed for development of an application for a web browser would take a lot less time. Javascript frameworks would be less bug-prone, easier to understand and would take up a fewer lines. The world would be a better, more accessible place. This, of course, is not going to happen in the near future (if ever). Even if projects like WaSP (the Web Standards Project) succeed in setting proper standards, there’s still the IE user running an older version that thumbs its nose at all those standards.

The browser as a platform for applications is still falling short: The user demands well-developed fast applications that can do it all. Be that as it may, but we’re pushing the current implementations of the concepts behind browser markup, presentation and interactivity to a maximum and it takes a lot of work to keep it all together.

Maybe I am the wrong guy for the browser-hell half of the job and maybe we need a proper web developer to help us out. One thing’s for sure: People who can do both web development and application development have a nice future in front of them.

To wrap it up: Without proper standards a vessel is needed to ensure a faux standard that allows people to keep on working without having to constantly worry about the difference in DOM, HTML and CSS implementations. And since the only common denominator between all those platforms that is at least somewhat standardized is the Javascript language, it has become the web’s little bitch.

What this means for OpenPanel

Once again: ExtJS is a fine framework with a lot of nice features and, again, it saved us a lot of time. But I think I’ll pass for now since we’ll start looking at the actual shortest route from A to B.

Since the functionality of the client is strictly separated into different concerns, it’s quite easy for me to write (and maintain) a working application while someone who is adept at browser differences is able to make things work without having to fully grasp how the application works. That way our application stays flexible, fast and lightweight. By the way, if you’re good at this stuff let us know because we’d love to have a little chat with you.

Bookmark It

When you are designing a project with complex storage requirements and some demands on reliability and performance, a few options come to mind.

Even though at OpenPanel we have strong feelings about NIH, we didn’t think writing our own database store was the way to go. Many smart people have already written many different database backends, which means that in both quantity and quality the database area is well-covered. So, writing our own was right out.

The second option that comes to mind is to use the most basic kind of database available: a key/value store like Berkeley DB or GDBM. Combining a few key/value stores together yields a lot of flexibility, but there’s a lot of glue you need to write, then. In programming language terms, the key/value API is not powerful.

The only way out then seems to be SQL. The common choice seems to be MySQL, and with good reason - it’s robust, fast, flexible (supporting most of the SQL standard), comes pre-packaged for any distribution, and just about any sysadmin or PHP-developer you run into knows his way around it more or less. A close second would be Postgres, less popular with the common PHP developer crowd, more popular with seasoned developers in other languages, and understandably so.

However, we did not go for the path well-traveled. After evaluating our requirements, we realized we barely needed the power of a client/server model database with high concurrency support. Also, we figured choosing a slightly less popular implementation would deter people from messing with the database by hand.

We chose SQLite. It’s extremely lightweight, reliable, robust and surprisingly fast. Choosing SQLite means our database is just a file (directory, to be honest) in /var/opencore, where users and admins can’t run into it by accident while mucking about in phpMyAdmin - but when they really need to mess with the database, they can, with their familiar SQL idioms.

SQLite has most the features a developer would expect from a database; transactions, subqueries, decent indexing support, triggers, and room for extension with user defined functions. The only thing sorely missing is foreign key support, but that’s easily implemented as a bunch of triggers.

Even though we now had this powerful database engine, we put in a lot of effort mapping our idea of an object revision model onto it (more about that in a later post) - but SQLite made it a lot less painful.

SQLite’s only real limitation seems its lack of concurrency, which is made worse by the locking model used that seems to invite polling for access instead of forming some kind of queue. For this reason, OpenPanel keeps just one handle to the database file and manages exclusive and shared access to it via the normal pthread mechanisms.

Incidentally, the current (unpublished) version of our automated web application installer tries to shoehorn its data into a key/value store and it’s hurting - even from Python! We’ll probably rewrite those bits to use SQLite as well (but, of course, separate from the OpenPanel database).

Summarizing, if you are looking for a data store for your software project, consider SQLite. It’s as lightweight as most key/value store libraries but throws in a hell of a lot more featurewise.

Bookmark It

People requested a screencast of our command line interface in action as well, so here it is:

We’ll be adding a dedicated screenshots section to our website soon.

Bookmark It

It all started as an enormous itch that needed serious scratching. We’ve been dealing, in our various capacities at hosting providers, with many incarnations of commercial server control panels like Ensim, Plesk and cPanel. Not only were these, in our view, haphazardly constructed kludges of magic that actively worked against the operating systems they were installed on; we kept coming back to the fact that people were actually paying for such software. Through the nose, no less. This felt wrong and paved the way for the OpenPanel project.

We spent over a year creating something with the capacity to change the landscape, by combining the proper amount of openness and flexibility with a state of the art user interface and a license that should make trumpets sound from the heavens. People we’ve shown it to have been raving. But, now that we’re at the point to hit the submit button for all this on sites like freshmeat, all those little doubts start setting in. It’s scary as hell.

I’m convinced we’ve taken the right step by not fully adopting the ‘release early’ mantra — it has given us the chance to quickly reshape the design where needed (Heck, we’re on the second incarnation of the GUI). But this is the downside. If you release early, expectations are lower and you never get this neckhair-raising fear of throwing your work into the world. Alas.

We’ve opened up the 0.9 (beta) branch of OpenPanel as of today. There’s no longer a requirement to email us to get access, just follow the download instructions from the main site. Messages have also gone out to the usual places. The future is now.

Bookmark It

Here’s a preview of the upcoming Beta release of our life’s work. Now that we’re over that “oh no, we’re all going to die and we’ll never finish it” point, things are starting to look well. Or perhaps the other way around:

We’ll pick up the last bugs we found monday and then seed a release candidate to our alpha team. If no show stoppers emerge, that will be followed up shortly by a full public release of the OpenPanel beta.

Bookmark It

Quick Update: We just finished documenting the bash-based shell API for opencore modules, so if you’re part of the alpha team and wanting to work on your own openpanel modules, read this PDF to get started.

Bookmark It