OpenPanel Development Blog

I’m a strong believer in systems that are easily discoverable, where the structure of the files that make up the system communicate something about the application’s internal structure to the outside world. I tried to do much of the same thing with OpenPanel, but at the end of the day there’s still a lot to go around, so perhaps now’s a good time to write up a bit about the thinking and architecture behind OpenPanel and its components.

When we started the design, we wanted to focus on the following demands:

1. OpenPanel should be modular, and modules should not be restricted to any specific language
2. There should be elementary protection against security defects in module code
3. There should be a possibility of multiple user interfaces (at the very least both a GUI and a command line tool)
4. The system should anticipate the possibility of remote control and clustered integration
5. Configuration provided by OpenPanel should follow the standard configuration practices for the programs involved — no funky business relying on access to a MySQL database
6. It should be possible to dedicate an installation to a single purpose (like mail) without requiring unrelated services (Apache, MySQL) through dependencies

What we came up with, is a design that could best be explained as a domain-specific object database that reflects changes into the configuration of outside programs. The opencore daemon implements this abstraction. At the bottom it has an sqlite database that stores object and class data. Modules are implemented as directories inside /var/opencore/modules that define object classes for the database and the program that executes the necessary changes to outside configuration related to objects of those classes. A JSON-based RPC interface accepts requests through HTTP. This backend port is also used to serve static files related to the web-based GUI.

Obviously, accepting remote RPC requests is something you don’t want to combine with root privileges. Also, since we wanted to keep a low number of privileges exposed directly to module code, running the back-end programs/scripts for module actions was definitely not something we felt should be done as the root user. This is where the authd daemon comes in. It accept requests on a unix domain socket that is only accessible to the opencore daemon. Whenever opencore runs a module program, it will fork to a new process, open a socket to authd, tell authd which module will be executed and drop the privileges needed to open any new connections to authd. The unprivileged module can talk to authd on fd 3, but will be restricted to those privileged actions defined in its module.xml file.

Documentation for the Module API and the RPC Interface is on our site, but some of it may still be a bit rough around the edges.

The problem with developing front-end applications for web browsers from a back-end developer’s point of view.

In our search for quick (and equally dirty) Javascript solutions for problems like form creation and control, we’ve run across a lot of frameworks. “Let’s use the framework which makes our lives easiest” was the chant, to implement anything third party loosely and, if it breaks, rip it out and either find something better or write something from scratch.

With that in mind we decided to go with ExtJS since it offered a lot of very useful widgets, the widest spectrum of features, decent API documentation and a big user base. Loose implementation of ExtJS’ FormPanel was easy: Create a FormPanel object, render it to an HTML element and forget it ever happened. The wow factor was high at first: All this without any hassle whatsoever, great. Now we didn’t have to worry about this anymore.

Apart from a few minor bugs and glitches, the forms worked fine. That is, until we ran some tests on Internet Explorer. The results kind of shocked us: ExtJS components inside a DIV appeared to have absolute positioning, ignoring the DIV’s overflow property. Also, long loading times and general slowness poured rain on our form parade.

Woes

Apparently there was some dodgy stuff going on inside the ExtJS framework, something intended to make sure it all looks and acts the same in all the major browsers. The long loading times are caused by the copious amount of (needed) ExtJS script infrastructure. Of course it’s still possible to minimize and strip a lot of unwanted script code, but since we were still in the process of developing an application we were not going to let ourselves be slowed down by something which could be postponed until a later stage.

The general slowness — especially in IE and Firefox — was caused by something completely different: If you take a look at the ExtJS API documentation you will find a lot of very elaborate objects. There’s a method for almost anything you want and if something’s missing, chances are you can access it through one of the event handlers. Great stuff. But since you’ll never need all of those nice features (and God forbid you will) you could do without most of it.

What we actually wanted was a form widget framework that performs the dirty footwork for us. We wanted it to be fast, flexible, skinnable and cross-browser. That’s what ExtJS promises. It’s there, it does work, but it’s slow and prone to failure. Since ExtJS is one of the most extensive frameworks around I think it’s rather safe to assume that any other similar widget frameworks will have comparable issues; Blaming this all on ExtJS is just too easy, but it’s safe to say that layering frameworks like ExtJS on top of the Javascript runtime will lead to a product that is rigid, huge and slow.

Barking up the right tree

So what’s really going on here? Why is ExtJS so rigid? What causes this slowness and bloat? Perhaps I’m just not competent enough to write an application for a web browser? Let’s go over the points:

• Rigid — lack of standards : Go quickly from A to B.
• Slow — weakness of Javascript : Go quickly from A to B.
• Huge — demands of industry : Go quickly from A to B.

Our positioning bug wasn’t actually caused by ExtJS proper, but rather by the fact that ExtJS needs to hack around a lot of browser-specific issues. We can go nuts over the fact that this problem exists in the first place, but hey, the fine people at ExtJS did their best to make it work anyway, so no, I don’t think I can point the bullshit ray exclusively at the ExtJS framework. The problem here lies in the lack of standards, caused by browser manufacturers being unable (or even unwilling) to cooperate with eachother.

The complexity of ExtJS objects puts a price on the general performance of the application: For every object there are tons of handlers, methods, properties and configuration options. Javascript was never meant to be used as an OO work horse for big applications, neither in architecture nor in performance.

The reason why the ExtJS framework is so big is that it has to serve many masters with many demands: There should be something for everyone and there are lots of people screaming for an easier way to write applications in Javascript. Is it way too bloated for what a browser can handle? I think it is. But who can blame them? I sure can’t. Using it saved us a lot of time and the demand for a framework like that is apparently big enough to justify its existence. So no, they’re trying their best to make something very decent and respect and credit where respect and credit are due.

Maybe I should just blame myself for being a lazy developer. I don’t really want to know why browser #1 does it like this and browser #2 does it like that. And to make it even worse: I don’t want to know how to fix or circumvent it. I think I’m a lot better off writing applications rather than fixing these kinds of inconsistencies. Does that make me lazy? One could argue so, but come on, imagine writing stuff in C and taking four different compilers into consideration. I think there are a lot of differences between web developers and application developers. For one, web developers usually aren’t anal retentive when it comes down to writing structured code. Application developers generally don’t know jack about CSS (and like it that way).

Conclusion

The lack of standards. IE does it this way, other browsers do it that way. If there would be proper standards life would be a lot easier. HTML and CSS would be more respected by both web developers and application developers. Mastering the skills needed for development of an application for a web browser would take a lot less time. Javascript frameworks would be less bug-prone, easier to understand and would take up a fewer lines. The world would be a better, more accessible place. This, of course, is not going to happen in the near future (if ever). Even if projects like WaSP (the Web Standards Project) succeed in setting proper standards, there’s still the IE user running an older version that thumbs its nose at all those standards.

The browser as a platform for applications is still falling short: The user demands well-developed fast applications that can do it all. Be that as it may, but we’re pushing the current implementations of the concepts behind browser markup, presentation and interactivity to a maximum and it takes a lot of work to keep it all together.

Maybe I am the wrong guy for the browser-hell half of the job and maybe we need a proper web developer to help us out. One thing’s for sure: People who can do both web development and application development have a nice future in front of them.

To wrap it up: Without proper standards a vessel is needed to ensure a faux standard that allows people to keep on working without having to constantly worry about the difference in DOM, HTML and CSS implementations. And since the only common denominator between all those platforms that is at least somewhat standardized is the Javascript language, it has become the web’s little bitch.

What this means for OpenPanel

Once again: ExtJS is a fine framework with a lot of nice features and, again, it saved us a lot of time. But I think I’ll pass for now since we’ll start looking at the actual shortest route from A to B.

Since the functionality of the client is strictly separated into different concerns, it’s quite easy for me to write (and maintain) a working application while someone who is adept at browser differences is able to make things work without having to fully grasp how the application works. That way our application stays flexible, fast and lightweight. By the way, if you’re good at this stuff let us know because we’d love to have a little chat with you.

When you are designing a project with complex storage requirements and some demands on reliability and performance, a few options come to mind.

Even though at OpenPanel we have strong feelings about NIH, we didn’t think writing our own database store was the way to go. Many smart people have already written many different database backends, which means that in both quantity and quality the database area is well-covered. So, writing our own was right out.

The second option that comes to mind is to use the most basic kind of database available: a key/value store like Berkeley DB or GDBM. Combining a few key/value stores together yields a lot of flexibility, but there’s a lot of glue you need to write, then. In programming language terms, the key/value API is not powerful.

The only way out then seems to be SQL. The common choice seems to be MySQL, and with good reason - it’s robust, fast, flexible (supporting most of the SQL standard), comes pre-packaged for any distribution, and just about any sysadmin or PHP-developer you run into knows his way around it more or less. A close second would be Postgres, less popular with the common PHP developer crowd, more popular with seasoned developers in other languages, and understandably so.

However, we did not go for the path well-traveled. After evaluating our requirements, we realized we barely needed the power of a client/server model database with high concurrency support. Also, we figured choosing a slightly less popular implementation would deter people from messing with the database by hand.

We chose SQLite. It’s extremely lightweight, reliable, robust and surprisingly fast. Choosing SQLite means our database is just a file (directory, to be honest) in /var/opencore, where users and admins can’t run into it by accident while mucking about in phpMyAdmin - but when they really need to mess with the database, they can, with their familiar SQL idioms.

SQLite has most the features a developer would expect from a database; transactions, subqueries, decent indexing support, triggers, and room for extension with user defined functions. The only thing sorely missing is foreign key support, but that’s easily implemented as a bunch of triggers.

Even though we now had this powerful database engine, we put in a lot of effort mapping our idea of an object revision model onto it (more about that in a later post) - but SQLite made it a lot less painful.

SQLite’s only real limitation seems its lack of concurrency, which is made worse by the locking model used that seems to invite polling for access instead of forming some kind of queue. For this reason, OpenPanel keeps just one handle to the database file and manages exclusive and shared access to it via the normal pthread mechanisms.

Incidentally, the current (unpublished) version of our automated web application installer tries to shoehorn its data into a key/value store and it’s hurting - even from Python! We’ll probably rewrite those bits to use SQLite as well (but, of course, separate from the OpenPanel database).

Summarizing, if you are looking for a data store for your software project, consider SQLite. It’s as lightweight as most key/value store libraries but throws in a hell of a lot more featurewise.

People requested a screencast of our command line interface in action as well, so here it is:

We’ll be adding a dedicated screenshots section to our website soon.

The premise sounds easy enough: There’s no arguing with free. In the server control panel market we found a niche dominated by commercial players with a stranglehold on price and a bunch of products that annoy system administrators to the bone. So we set out to create something appealing — software with disruptive potential. With a product in our hands that ticks all the checkboxes in terms of nerd appeal, you would say that this would be as easy a sell as free blowjobs at a LAN party. Unfortunately, things are not quite that simple.

We went through the hoops submitting the news of our release to a dozen of the usual suspects, got a couple of hits and the first testers have arrived. There’s even a couple of people around doing more involved things like trying to port packages to other distros. It’s not bad and we appreciate all the people helping us out by playing with the beta, but a stampede it is not. Building a community takes time.

In terms of strategy, this may be the real reason for the ‘release early, release often’ mantra. Especially when your project is in a niche area, expect community building to take up maybe just as much time as actual development. We wanted to avoid getting distracted by actual users too early in the game, so our gut feeling at the time was ‘early, schmearly’. We had a couple of other reasons, some of them even sounded real good — like how there are too many open source projects in the field that need so much polishing that the market has been mostly ignoring them. But the fact of the matter is, all the hours of community building we’ve been avoiding up to the beta release will have to be put in double now. We’ll need to go from our current mode of reclusive introversion to full-blown extravert. Tough job for a bunch of nerds.

It all started as an enormous itch that needed serious scratching. We’ve been dealing, in our various capacities at hosting providers, with many incarnations of commercial server control panels like Ensim, Plesk and cPanel. Not only were these, in our view, haphazardly constructed kludges of magic that actively worked against the operating systems they were installed on; we kept coming back to the fact that people were actually paying for such software. Through the nose, no less. This felt wrong and paved the way for the OpenPanel project.

We spent over a year creating something with the capacity to change the landscape, by combining the proper amount of openness and flexibility with a state of the art user interface and a license that should make trumpets sound from the heavens. People we’ve shown it to have been raving. But, now that we’re at the point to hit the submit button for all this on sites like freshmeat, all those little doubts start setting in. It’s scary as hell.

I’m convinced we’ve taken the right step by not fully adopting the ‘release early’ mantra — it has given us the chance to quickly reshape the design where needed (Heck, we’re on the second incarnation of the GUI). But this is the downside. If you release early, expectations are lower and you never get this neckhair-raising fear of throwing your work into the world. Alas.

We’ve opened up the 0.9 (beta) branch of OpenPanel as of today. There’s no longer a requirement to email us to get access, just follow the download instructions from the main site. Messages have also gone out to the usual places. The future is now.

Here’s a preview of the upcoming Beta release of our life’s work. Now that we’re over that “oh no, we’re all going to die and we’ll never finish it” point, things are starting to look well. Or perhaps the other way around:

We’ll pick up the last bugs we found monday and then seed a release candidate to our alpha team. If no show stoppers emerge, that will be followed up shortly by a full public release of the OpenPanel beta.

A major source of anger and frustration in C++ style OO is multiple inheritance. It’s a source of anger and frustration and one most people recognize as a path best avoided. The Objective-C idea of a protocol is a real life-saver in many occasions where you would need to deal with multiple inheritance otherwise. The concept of a protocol is to have a second way of classifying objects, one that completely sidesteps the hierarchical class model and instead just classifies objects by their common functions.

One area where the concept of protocols is quite convenient is iteration. There is usually a wide area of possible classes that could, theoretically, enumerate a list of contained items. Although C++ lacks protocols proper, template iterators are a fine way to access any class that implements a de facto iterator protocol. The only thing missing is an explicit declaration of the implemented protocol.

The Grace iterator<collectionclass,nodeclass> template is a minimized version of the visitor<> template. It requires one function (visitchild) to be defined inside your class that returns a pointer to a child node. Here is what it would look like for a purely synthetic class:


class syntheticlist
{
public:
   syntheticlist (void) {}
   ~syntheticlist (void) {}
   string *visitchild (int atpos)
   {
       if ((atpos<0)||(atpos>1)) return NULL;
       if (atpos==0) str = “Hello”;
       else str = “world”;
       return &str;
   }
protected:
   string str;
};

int myApp::main (void)
{
   syntheticlist L;
   foreach (n,L) fout.writeln (n);
   return 0;
}

The foreach macro goes through the following steps:

1. It creates an iterator<syntheticlist, string *> pointing to L
2. It sets up a for loop that starts at visitchild(0) and stops when it returns NULL
3. It creates a temporary reference variable n linked to the current node.

The Grace iterator protocol looks a bit ascetic with only visitchild and the lack of the traditional first() and next() pattern. Part of this is due to the visitor protocol (which iterator is a part of) being about more than just of iteration. If your goal, however, is to make your class iterable with foreach, you can safely assume an argument value 0 to mean ‘first’ and any value bigger than 0 to mean ‘next’ if that makes more sense in your context. Most collection classes used within Grace use growable arrays, which means they don’t have to keep any state information in order to be iterable.

Next to string and array handling, memory management has always been one of those reasons users of other languages pointed at C++ programmers and laughed. The problem stems from dealing with temporary objects and keeping track of their lifetime. Although remembering to delete objects after you’re done with them sounds easy on paper, in the real world it’s tedious and you’re always just one forgetful early return statement away from a memory leak.

A lot of modern languages offer a friendlier form of memory management for this reason. They keep track of all references to a particular object and regularly take care of objects that are no longer referenced. This garbage collection process has a number of drawbacks that are usually seen as justified, considering the programming problems it solves. And it does solve them, but not all problems it solves need to be problems and, once you take those non-problems off the list, the smell of over-engineering becomes apparent.

What I see as the most interesting memory management problem that garbage collection solves can be illustrated by the following snippet:

Storpel *createStorpel (int modelNumber)
{
   Storpel *res = new Storpel;
   res->setModel (modelNumber);
   return res;
}

int main (void)
{
   Storpel *myStorpel = createStorpel (42);
   myStorpel->grind (11);
   delete myStorpel;
   return 0;
}

The problem with this pattern is obvious: The receiving function, by calling createStorpel() has taken responsibility for managing the Storpel object. This can get hairy in almost zero time.

In Grace, I prevented this anti-pattern from emerging by sticking to a number of principles:

• Use of pointers and new for objects is minimalized (pointers are awkward when you want to use overloaded operators anyway, so there’s exterior motivation).
• Data container classes are designed to be mere vessels for a raw data block that can be taken away from one object and ‘given’ to another without copying.
• When confronted with a pointer during assignment and initialization, data container classes will ‘take over’ the other object’s data and immediately delete it.
• Functions that want to return objects actually return pointers to temporary objects. The returnclass (type) name retain macro creates such a temporary object plus a temporary reference to make it easier to access the object.

Words are words, code is code. This is what it effectively looks like:

string *getGreeting (void)
{
   returnclass (string) res retain;
   res = “Hello, world.”;
   return &res;
}

int main (void)
{
   string s = getGreeting();
   fout.writeln (s);
   return 0;
}

Behind the curtains, the returnclass macro creates a temporary string object using a custom allocator. This string object creates a persistent refblock structure to hold the data. Then, inside main(), we assign it to s:

The receiving object takes a reference to the underlying refblock:

And then the original object is immediately destroyed:

When main() goes out of scope, the ’s’ string object will be automatically cleaned up and, with it, the refblock. No need for an explicit delete.

As for the other problems that garbage collection solves, most of them have to do with interactions between collections of more complex objects. Mostly, if you find yourself in need of tracking such complex relationships, you may need to look at a way of simplifying your design. If that is not possible, then keeping track of memory references takes no more diligence than what the structure already demands.