OpenPanel Development Blog

I’m a strong believer in systems that are easily discoverable, where the structure of the files that make up the system communicate something about the application’s internal structure to the outside world. I tried to do much of the same thing with OpenPanel, but at the end of the day there’s still a lot to go around, so perhaps now’s a good time to write up a bit about the thinking and architecture behind OpenPanel and its components.

When we started the design, we wanted to focus on the following demands:

1. OpenPanel should be modular, and modules should not be restricted to any specific language
2. There should be elementary protection against security defects in module code
3. There should be a possibility of multiple user interfaces (at the very least both a GUI and a command line tool)
4. The system should anticipate the possibility of remote control and clustered integration
5. Configuration provided by OpenPanel should follow the standard configuration practices for the programs involved — no funky business relying on access to a MySQL database
6. It should be possible to dedicate an installation to a single purpose (like mail) without requiring unrelated services (Apache, MySQL) through dependencies

What we came up with, is a design that could best be explained as a domain-specific object database that reflects changes into the configuration of outside programs. The opencore daemon implements this abstraction. At the bottom it has an sqlite database that stores object and class data. Modules are implemented as directories inside /var/opencore/modules that define object classes for the database and the program that executes the necessary changes to outside configuration related to objects of those classes. A JSON-based RPC interface accepts requests through HTTP. This backend port is also used to serve static files related to the web-based GUI.

Obviously, accepting remote RPC requests is something you don’t want to combine with root privileges. Also, since we wanted to keep a low number of privileges exposed directly to module code, running the back-end programs/scripts for module actions was definitely not something we felt should be done as the root user. This is where the authd daemon comes in. It accept requests on a unix domain socket that is only accessible to the opencore daemon. Whenever opencore runs a module program, it will fork to a new process, open a socket to authd, tell authd which module will be executed and drop the privileges needed to open any new connections to authd. The unprivileged module can talk to authd on fd 3, but will be restricted to those privileged actions defined in its module.xml file.

Documentation for the Module API and the RPC Interface is on our site, but some of it may still be a bit rough around the edges.

Bookmark It

When you are designing a project with complex storage requirements and some demands on reliability and performance, a few options come to mind.

Even though at OpenPanel we have strong feelings about NIH, we didn’t think writing our own database store was the way to go. Many smart people have already written many different database backends, which means that in both quantity and quality the database area is well-covered. So, writing our own was right out.

The second option that comes to mind is to use the most basic kind of database available: a key/value store like Berkeley DB or GDBM. Combining a few key/value stores together yields a lot of flexibility, but there’s a lot of glue you need to write, then. In programming language terms, the key/value API is not powerful.

The only way out then seems to be SQL. The common choice seems to be MySQL, and with good reason - it’s robust, fast, flexible (supporting most of the SQL standard), comes pre-packaged for any distribution, and just about any sysadmin or PHP-developer you run into knows his way around it more or less. A close second would be Postgres, less popular with the common PHP developer crowd, more popular with seasoned developers in other languages, and understandably so.

However, we did not go for the path well-traveled. After evaluating our requirements, we realized we barely needed the power of a client/server model database with high concurrency support. Also, we figured choosing a slightly less popular implementation would deter people from messing with the database by hand.

We chose SQLite. It’s extremely lightweight, reliable, robust and surprisingly fast. Choosing SQLite means our database is just a file (directory, to be honest) in /var/opencore, where users and admins can’t run into it by accident while mucking about in phpMyAdmin - but when they really need to mess with the database, they can, with their familiar SQL idioms.

SQLite has most the features a developer would expect from a database; transactions, subqueries, decent indexing support, triggers, and room for extension with user defined functions. The only thing sorely missing is foreign key support, but that’s easily implemented as a bunch of triggers.

Even though we now had this powerful database engine, we put in a lot of effort mapping our idea of an object revision model onto it (more about that in a later post) - but SQLite made it a lot less painful.

SQLite’s only real limitation seems its lack of concurrency, which is made worse by the locking model used that seems to invite polling for access instead of forming some kind of queue. For this reason, OpenPanel keeps just one handle to the database file and manages exclusive and shared access to it via the normal pthread mechanisms.

Incidentally, the current (unpublished) version of our automated web application installer tries to shoehorn its data into a key/value store and it’s hurting - even from Python! We’ll probably rewrite those bits to use SQLite as well (but, of course, separate from the OpenPanel database).

Summarizing, if you are looking for a data store for your software project, consider SQLite. It’s as lightweight as most key/value store libraries but throws in a hell of a lot more featurewise.

Bookmark It

It all started as an enormous itch that needed serious scratching. We’ve been dealing, in our various capacities at hosting providers, with many incarnations of commercial server control panels like Ensim, Plesk and cPanel. Not only were these, in our view, haphazardly constructed kludges of magic that actively worked against the operating systems they were installed on; we kept coming back to the fact that people were actually paying for such software. Through the nose, no less. This felt wrong and paved the way for the OpenPanel project.

We spent over a year creating something with the capacity to change the landscape, by combining the proper amount of openness and flexibility with a state of the art user interface and a license that should make trumpets sound from the heavens. People we’ve shown it to have been raving. But, now that we’re at the point to hit the submit button for all this on sites like freshmeat, all those little doubts start setting in. It’s scary as hell.

I’m convinced we’ve taken the right step by not fully adopting the ‘release early’ mantra — it has given us the chance to quickly reshape the design where needed (Heck, we’re on the second incarnation of the GUI). But this is the downside. If you release early, expectations are lower and you never get this neckhair-raising fear of throwing your work into the world. Alas.

We’ve opened up the 0.9 (beta) branch of OpenPanel as of today. There’s no longer a requirement to email us to get access, just follow the download instructions from the main site. Messages have also gone out to the usual places. The future is now.

Bookmark It

The Grace value class is a pretty useful container for tree-data. It walks and quacks like an associative array that, combined with the foreach macro makes it easy to inspect and manipulate tree structures using compact code like this:


void printListOfPersons (const value &persons)
{
 foreach (p, persons)
 {
   fout.writeln (”%s <%s>” %format (p["name"], p["email"]));
 }
}

Getting data into such a structure posed to be a bit of a challenge, though. You ended up with huge blocks of code with lots of repetitions:

persons["john"]["name"] = “John Smith”;
persons["john"]["email"] = “jsmith@example.net”;
persons["steve"]["name"] = “Steve McSmith”;
persons["steve"]["email"] = “steve@example.net”;

The repetitive nature wasn’t only straining on the hands, the build-up here also involved looking up the node for ‘john’ and ’steve’ twice. Using the negative array index to refer to the last node was a nice timesaver in this respect:

persons["john"]["name"] = “John Smith”;
persons[-1]["email"] = “jsmith@example.net”;

The problem with this approach, apart from it still being pretty repetitive, is that it starts you thinking that the idiom persons[-1] means “the node I created in the line before”, but there are lots of situations where you cannot really know this. Take this example:

persons = listPersons();
persons["john"]["favoriteColor"] = “blue”;
persons[-1]["favoriteSwallowType"] = “African”;

All it takes for this code to break is the inclusion of the key ‘john’ in the result set of listPersons() at a position that is not the last one in the array. This may sound like a mistake that is easy to avoid, but in an application that sees a bit of growth, the assumptions you make in this respect may stop being valid in the future, and you’ll have forgotten all about them.

The solution to this problem came to me as an inspiration from jQuery’s clever discovery of this one symbol in the C-like namespace that managed to remain completely overlooked by most library builders: The dollar sign. On the global level, two variations of the function $(…) are defined that create a retainable pointer to a value object:

• $(”value”) creates an array and adds a node with “value” as its value data.
• $(”key”,”value”) creates a dictionary and adds a node with the key “key” and “value” as its value data.

These functions create a pointer to a value object, so the library also defines value::$(…). Now you can chain nodes together, like this:

value v = $(”john”,
             $(”name”, “John Doe”) ->
             $(”email”, “jdoe@example.net”) ->
             $(”favoriteColors”,
                 $(”Titanium White”) ->
                 $(”Cadmium Yellow”)
              ) ->
             $(”favoriteNumber”, 42)
          ) ->
         $(”steve”,
             $(”name”, “Steve Jibs”) ->
             $(”email”, “sjibs@example.net”) ->
             $(”favoriteColors”, $(”Black”)) ->
             $(”favoriteNumber”, 1)
          );

The result is pretty compact on a scale from 1 to JSON. There are some extra functions that will come of use:

• $attr(key,value) Sets an attribute.
• $type(type) Sets the value’s type().
• $val(otherval) Sets the value’s data only to that of otherval.
• $merge(otherval) Merges the child nodes and attributes of another value.

After these functions were added to grace, I started using them religiously. Inline declarations are a powerful way to keep your code readable. Thanks, jQuery.

Bookmark It

Combining POSIX threads and the fork() system call can be a bit messy. Here’s some facts you may not be aware of:

1. A fork() call only clones the current thread
2. This can have unintentional effects

The most noticable effect of this, for aspiring thread users, is that it’s unsafe to start any threads before doing the daemonization double fork().

A less obvious problem, though, is that any locks that are held by threads other than the current thread will remain in a locked state within the child’s context until the heat death of the universe. In situations where there are global background objects that keep locks, this means that doing much beyond an exec() is unsafe and, if that exec() fails, it is unsafe to call exit() if any global destructors touch locks. Use _exit() instead.

Bookmark It

Quick Update: We just finished documenting the bash-based shell API for opencore modules, so if you’re part of the alpha team and wanting to work on your own openpanel modules, read this PDF to get started.

Bookmark It

Hi!

Last week we released new alpha packages to our group of testers, both .deb and .rpm packages. See the changelog for a list of what’s new and great!

Bookmark It

First of all, what is up with the Debian 2.4 kernel? I’m glad that Etch will default to 2.6, but the 2.4 kernel that has been with Debian for so long is just plain Ghetto. Running multithreaded applications on a non-NPTL kernel is atrocious. But, on a positive note, running into this trailer park kernel on a machine that was upgraded to Etch from an earlier version, all this did point me to an interesting tidbit about using setuid and friends: If you want to use these to drop privileges, make sure you do this before you start any threads or madness will ensue.

Bookmark It

Grace uses pthread conditionals to facilitate communication between threads. Worker threads waiting for new events can either wait indefinitely for an event or sleep on it for a determined number of milliseconds. The latter is implemented using the pthread_cond_timedwait() call. And this is where design-by-committee has gone completely insane. Someone thought he was being clever and specified the resolution for this call to be set in nanoseconds. Just what you need if you want to trigger conditionals in your particle accelerator. A bit of a bummer, though, if you want to use this call in, you know, actual real world conditions.

(more…)

Bookmark It

Historically, the Grace library has been really forgiving, if not lax, in parsing XML formats. In an ideal world, every document is well-formed, but sometimes it’s a seller’s market (take RSS feeds) and you have to work with what you get. Although this worked out well in general, we were having a lot of problems with less-than-well-formed documents every time a new module was checked in. Each OpenPanel module comes with an XML file that contains information about the object classes it represents, their relationships to other classes and their impact on the user interface. It’s easy to make a mistake here, but hard to understand what goes wrong if things don’t work out and you get unhelpful errors.

(more…)

Bookmark It