29Apr/10Off

Distribution is Engineering Too

by Dan

In the last post I announced a series on CULA's engineering philosophy. And the first post in our series on engineering philosophy is. . .distribution? That doesn't seem very engineering oriented. Why would we start here?

We choose to lead off our series with distribution for a variety of reasons. For starters, without distribution, there would be no product. Well, there might be a product, but you wouldn't be able to get it. Which I'd say is as good as no product at all. Secondly, although the details of our distribution policies are guided by business decisions, the implementation of these policies turns out to be a very technical task. Recently, these technical tasks came up front and center when we had a problem with the distribution of one CULA 1.3's packages. We wanted to use this time to explain how this problem came about and what we did to fix it, not because it's necessarily relevant to CULA or GPUs or linear algebra, but just because we found it interesting.

So here we are, starting off with a discussion on distribution. When we released CULA, we split our capability set into Basic and Premium versions, and we needed a distribution mechanism that could handle these differences. The distribution mechanism required many different parts, including a user database, purchase tracking system, and download manager, all of which needed to interoperate in a manageable way. We used many different off-the-shelf components when building our system, but wrote custom code to tie these components together. Which brings us back to our CULA 1.3 release, where a problem in the way that these different parts were interoperating prevented users from downloading our 64-bit Linux package.

We never had a download problem before and this one seemed to come out of nowhere. As we began to unravel the long chain of tools between the download manager and the user, including our CMS, PHP, Apache, and our webhost's servers/configuration, we became increasingly convinced that this problem was going to be very difficult to diagnose and fix. After many hours of debugging and several calls to our hosting provider's tech support, the bottom line was that between the download manager and our CMS, a set of incompatible HTTP headers were issued that caused some security component (in Apache ... we think) to rewrite the download length to zero, preventing a user from successfully downloading.

Oddly, this error only happened if a file's size was greater than 16 MB. We found that this problem was unique to the download manager we had chosen, but unfortunately also found that the problem wasn't fixable in the code for that module. Given this, our options were to integrate a different download manager, to make edits in our CMS (making it hard to update with new security fixes later), or to roll our own. We choose to roll our own, and in doing so we cut out several of the extra layers of complexity, fixed the download problem, and ended up with a design that we think is better than the one we started with.

So, I guess that leads to one more technical aspect of distribution. If you're in a small but growing company, like us, you might find that despite your mastery of computer architecture and GPU programming, you'll be doing web programming from time to time. We each wear many hats, but we always bring a technical perspective to whatever we do, including this blog.

Comments (0) Trackbacks (0)

Sorry, the comment form is closed at this time.

Trackbacks are disabled.