Anyway.  At work, I was looking at cleaning up one database table of a project I'm working on, and I noticed that we have three ways that we are storing boolean values in the table:

• unsigned tinyint, which presumably would only be set to 0 or 1
• char(1), which also should be set to 0 or 1
• enum('y',n')

I, personally, always prefer the tinyint route.  Not really for a technical reason so much as a historical one ... it's just kind of the first one I picked.   What I would really like is if MySQL had a *real* boolean type field similar to postgresql, where the values can be TRUE, FALSE, 't' or 'f'.  It makes things so much easier.

MySQL will accept BOOL as a column type when creating a column, but it's implementation is a bit jacked in my opinion.  It creates an unsigned tinyint column, with null attributes.  That just gives a huge range of possible options, and doesn't really come that close to a binary option set at all.

mysql> create table test (steve bool);
Query OK, 0 rows affected (0.00 sec)

mysql> desc test;
+-------+------------+------+-----+---------+-------+
| Field | Type       | Null | Key | Default | Extra |
+-------+------------+------+-----+---------+-------+
| steve | tinyint(1) | YES  |     | NULL    |       |
+-------+------------+------+-----+---------+-------+
1 row in set (0.00 sec)

I did a bit of research, since enum seems now like the most reasonable option -- it limits you to a strict sub set of options.  The only question I have is, how well would that index?  Would it be faster scanning the table for enums or integers?  That's where I'm not sure.  It turns out that an enum that stores up to 255 values will only use 1 byte (assuming I'm intrepreting this MySQL reference book correctly).  A tiny integer uses the same size.  So it seems like they should both be pretty optimal, but I dunno.

Any thoughts?

So, the order would be something like: ' ', ' ', '1', '2', '3', 'A', 'B', 'C'.

What I really wanted was to display the blank ones last, since I wasn't interested in those. I poked through the string functions available to see if I could conjure up a hack, and ASCII works great, as it fetches the ASCII numeral of the first character in the string. And, if the string is empty, it will return a zero. And that's all I needed, was a binary flag to order by first.

Here's a sample query then:

SELECT string FROM table ORDER BY ! ASCII(string), string;

And the result would be: '1', '2', '3', 'A', 'B', 'C', ' ', ' '.

Perfect.

Here's the general principles I already plan on covering: foreign key support, data types, transactions, shell interface, ANSI SQL support, table types, general features, history, licensing, abstraction layers (using PHP).

Also, and I don't mean to sound like I'm spreading FUD, but it occurred to me this morning that I've never heard anyone say that MySQL is better than PostgreSQL.

Anyway, ideas welcome. I'll post my progress as I get the paper put together. This is something I've been meaning to do for a long time.

I haven't used it yet, but I had heard it was coming in PHP 5 for a while. Personally, I've always used PEAR::DB and was quite happy with that.

I'm still not sold on using the new layer anyway, but I figured I'd do some reading while I am getting ready to use it in this very small instance that I'm implementing.

On the docs page, I found a great summary of why prepared statements and stored procedures are handy and helpful. In short: they save you time for queries you have to repeat a lot, by pre-compiling the preparation that is common to all the queries, so that the database is really only processing the new data, and thus using less resources.

Prepared statements I haven't played with much before until a few weeks ago, but I've slowly started using them in my import scripts. Performance-wise, I've only seen about a 15 to 20 percent speed increase. The thing I like the most about them, though, is that I don't have to escape my strings anymore. That's a nice little advantage I can live with.

Anyway, php.net's PDO documentation page has a nice writeup as well, and instead of trying to summarize it myself any more, I'll just quote it verbatim:

Many of the more mature databases support the concept of prepared statements. What are they? You can think of them as a kind of compiled template for the SQL that you want to run, that can be customized using variable parameters. Prepared statements offer two major benefits:

• The query only needs to be parsed (or prepared) once, but can be executed multiple times with the same or different parameters. When the query is prepared, the database will analyze, compile and optimize it's plan for executing the query. For complex queries this process can take up enough time that it will noticeably slow down your application if you need to repeat the same query many times with different parameters. By using a prepared statement you avoid repeating the analyze/compile/optimize cycle. In short, prepared statements use fewer resources and thus run faster.
• The parameters to prepared statements don't need to be quoted; the driver handles it for you. If your application exclusively uses prepared statements, you can be sure that no SQL injection will occur. (However, if you're still building up other parts of the query based on untrusted input, you're still at risk).

I should mention that I'm linking to it because I agree with the author and also because I'm biased towards PostgreSQL. I prefer postgres not because of fanboyism, but because of experience and years of using both databases.

I was actually lucky enough to be trained to use PostgreSQL as the first database I ever used, and everything after that has never been able to duplicate its feature set. Since my first tech job, I've worked with Access, MySQL, SQL Server 2000 and SQLite.

Anyway, I love postgres. If you've never given it a chance, and you are looking for more advanced features, check it out. It's all that and a box of girl scout cookies. I tell you what.

The major change was that I added a new table for the chapters. It seems a little odd having the chapter number in a table all its own, but for a normalized database schema it makes perfect sense. The only thing I don't like is now you have to INNER JOIN across four tables just to get all the information. Most of the time you won't need anything but book + chapter + verse, which is only three tables. I did create a sample view called view_verses which pulls them all together so you can easily run a select on some format like 'Gen 1:1'. The thing I don't like is that even that view is CPU intensive, so I may have to look at changing some stuff around.

Aside from that basic view, I've decided I'm not going to put all my fun ideas for functions and views in the packaged release. Instead, I'll just have them either as a separate release, or just post them on the website since I'm sure they will evolve.

One really cool thing about postgres that I love is that you can have overloaded functions. I started playing with them a while back on this database, and came up with some cool concepts. One idea I want to implement is being able to run a select statment using a between on two verses. An example query would be: "SELECT * FROM view_verses WHERE verse_id BETWEEN verse('Gen.', 1, 5) AND verse('Genesis', 12); where the verse() function would be overloaded to take between one and three arguments: book, chapter and verse.

It's pretty cool all the stuff you can do with postgres, and that's definately where I'll be focusing my attention in getting the goodies done.

Anyway, if you want to download this test schema, its available here. As always, feedback is welcome.

GPNL is meant to be a QA tool for treecleaners to use, making it easier to find packages and ebuilds that ... well, need some lovin.

Though it's primarily intended for quality assurance, I've written the frontend to be hopefully pretty generic so anyone can browse the portage tree and just see some interesting statistics all around. There's still a lot more to be done on the website, but I think it's to a point right now where it's at least ready for some public consumption.

One thing I'm excited about is setting up the advanced search page, where you'll be able to run all kinds of funky queries. I'm going to be adding some more QA checks as well, once I get some time. Getting this much done though was quite a lot of work though, and I'm probably going to take a break and focus more on other things for a while. However, if anyone has some reasonable feature requests, I'm all ears.

Oh, also the source code for the database schema and the import scripts is available online. I'll setup SVN access and some documentation on the db layout sometime soon, not to mention how to get it working (short howto: emerge php, pkgcore, postgresql and portage-utils).

Also, a huge shout out to marienz and ferringb who put together pkgcore and my little python scripts that made importing the data incredibly simple. Thanks, guys.

The way to pull them out is by running "SELECT CONVERT(VARCHAR, GETDATE(), @x);" where @x is a positive integer. If you can find the right integer, you can save time and pull out something directly like '11-10-2006' as your variable.

One of the problems you'll run into though is that you can't just do 1 through $integer. Only some of them return something, and the ones that don't just throw an SQL error, so you get to hunt down which integers return something.

Well, digging for them manually once is something I don't want to repeat, so I wrote a query statement to pull out some of them. This could be a handy reference inside your database somewhere.

DECLARE @x int;
SET @x = 1;
WHILE @x < 132 BEGIN
IF @x > 0 AND (@x < 14 OR (@x > 20 AND @x <25) OR (@x > 99 AND @x < 115) OR @x IN(126,130,131)) BEGIN
SELECT @x, CONVERT(VARCHAR, GETDATE(), @x);
END
SELECT @x = (@x +1);
END

Here's another online reference you can use.

In working on getting portage into postgres, part of the problem I'm trying to solve is find out where QA issues are so they can be fixed. Unfortunately, in the early stages of my little script, it always assumes that the everything is working correctly across the board, so I'll write my queries assuming the foreign keys won't break. In reality, that doesn't happen, and I end up killing a transaction with hundreds of thousands of statements because there's 25 queries that break.

So, I had to write a postgres function to do the checking for me. This is going to be absolutely boring to those db gurus out there, but this is still slightly new to me, so I'm really enjoying it.

DECLARE
use_id integer;
BEGIN
SELECT id FROM use WHERE name = $2 AND
(package IS NULL OR package = $3) LIMIT 1 INTO use_id;
IF use_id IS NOT NULL THEN
INSERT INTO ebuild_use (ebuild, use) VALUES ($1, use_id);
RETURN TRUE;
ELSE
RETURN FALSE;
END IF;
END;

A really simple function, I know. All it does is run a SELECT statement to see if my foreign key is going to break *before* running the INSERT statement. That way, I can continue on my happy way with my transaction, and at the same time, if I want to turn on 'qa mode' when inserting the data, I can check for a false return on the recordset, and know which ebuilds need attention.

Pretty cool stuff, I think. Also, for the record, pgaccess is a *great* little GUI tool to quickly and easily edit your functions.

I've been working on getting everything about the portage tree into postgresql so you can run all kinds of queries. What kinds of queries? How many ebuilds use eclass 'eutils' and USE flag 'alsa' and are in 'video' herd and amd64 is masked but x86 isn't. That kind. Funky ones.

I must say, I really love postgresql even though I haven't been using it regularly for a long time, I'm quickly getting back into it. The simplicity, the standards, the power, the tools ... postgres has it all. Ahh, fanboyism.

Anyway, getting the details of the ebuilds was made incredibly easy thanks to marienz and ferringb and their work on pkgcore (and a custom python script). After that, it was just a matter of parsing the information and setting up the schemas. My importer is written in PHP and the class to import / read the data is still in its slightly butt-ugly stage. It can use some cleaning up, for sure. The database layout is going to be where the real optimizations are though. I'm going to work on setting up some good views so it will be easy to query. Right now, here's the list of tables I have setup: arch, category, ebuild, ebuild_arch, ebuild_eclass, ebuild_homepage, ebuild_license, ebuild_use, eclass, eclass_use, herd, license, package and use. All of them can already be populated by the scripts except for eclass_use and herd. I haven't setup the dependency ones yet, though that'll be pretty simple too.

So there's my big announcement. Woots. I'm working on creating the SQL to import everything right now (which takes a long time), and once that's done, I'll throw up a db dump somewhere. There's still lots to be done, like finishing the import scripts and setting up some webpages to browse the tree, but it shouldn't be too hard. I'm definately over the worst of it.

You want to know the way to *really* speed up your database? Normalization is probably going to be the largest factor. After that, use views, stored procedures, indexes, transactions and well-written queries, and your database is going to fly amazingly fast.

At work we have a large server we call "Zeus" because it is incredibly large. I won't even go into specs because you wouldn't believe me even if I told you. When I first started working here, the database running on it was incredibly slow. At first I blamed it all on the database software we are using (you can search my blog if you really wanna know which one it is. Hint: it's neither of the two mentioned above), but as we cleaned up the databases and tables by removing columns that were complete cruft and then doing everything I mentioned above, this puppy flies. In fact, our "dev" database, which is running on nothing more than an Athlon XP 1800+ runs just as fast as our beast-monster does.

That's how you get a fast database -- doing things the right way. Who would have thought?

I have to apologize for the elitist feel of this post, but my point is this ... the only magic bullet in improving performance is going to be quality code and design. Just replacing your database with something else isn't going to make the speed fairy sprinkle your application with love.

My beef this time around is that SQL Server 2000 doesn't have a TRIM() function. What's even more strange, is that they do have RTRIM and LTRIM functions. So, if you want to trim something you just run "UPDATE table SET foo = RTRIM(LTRIM(foo));" Brilliant!

Just another reason to love open source software -- you get incremental bug fixes and feature upgrades. This would have been fixed in any other database a long time ago, but since new features == new versions released every five years with Microsoft, you're just plain stuck with what you paid for the first time around.

Yes, I know I'm trolling but really ... what defense do they have? I know I can create a UDF to do the work for me, but that's besides the point that I'm right and prefer laziness. Plus, it's one thing to say "MS suxxors" and another to point out why their software is so shoddy. And if there's one thing I enjoy doing, it's educating people on why they're wrong and Gentoo is the extreme answer to everything.

Now go emerge postgres, ya jerk.

At work we use databases much, much more than I ever have anywhere else. Because of that, I've been forced to stretch and learn more ways to do things efficiently because if things are going slow, we are losing $$$. All that's nice and good, but the part I'm enjoying as a developer is the convenience of that views, stored procedures and user-defined functions give me.

For instance, I'll use a recent example. We have three tables that we pull data from a lot, and I'm constantly doing INNER JOIN foo ON bar = poop across a lot of pages. Well, a view lets me take the most critical data of those three tables and lump them all into one subset. So now, I can just do 'SELECT * FROM view-la-la WHERE foo-poo-pah' and be done with it.

I really have no idea how I lasted so long without these great features. It's probably because I was stuck using that other open-source database that has said for years that those features weren't important. Well, that and because sometimes I won't try something new if I'm familiar with what I have. Until they stop selling my favorite flavor of Cream of Wheat at the store, and then I *have* to try something different. Mmm... Malt-O-Mealie.

These new cartoons are a lot grittier than even Batman: The Animated Series was. The animation isn't as sharp, though, but it is detailed in other places. It's a trade-off, I guess.

Anyway, here's my impressions so far:

Oracle XE

• Easy to install
• A pain in the arse to connect, what the crap is TNS and SID variables I need to set? I'm really lost.
• The web interface is really slick.
• The instant client installed without any issues on Gentoo (on both 32 and 64 bit installs, I dual boot).

SQL Server 2005 Express Edition

• Just as easy to install
• No cool frontends, no nothing. It does come with a separate SQL script thing you can run in DOS (that's odd).
• I can't figure out for the life of me how to connect with PHP, but I don't think it's going to be easy or possible, considering its so new. ODBC *might* work.
• I'm not even sure if its accepting remote connections yet. I can't even get an ODBC connection created.
• I would be using the 2000 version, but I didn't find it til much later, and I've already got everything installed.
• I would have found it much earlier if Microsoft didn't make it so freaking absolutely gouge-my-eyes-out incredibly hard to find. Naming the databases 2000 and 2005 doesn't help my search queries too much. Not that sql or server are really original either. Come up with some original names, guys, like ... "Relational Asinity" or something.

Firebird

• I can't even get the thing started or figure out how to create a database.
• I tried the web admin (ibwebadmin), and while it looks like a really cool app, see above.
• Couldn't find any cool foss guis, so I gave up after searching for just a little bit to try the other ones.

IBM DB2

• Just barely finished installing it, I'm feeling optimistic about this one.
• Not too excited about installing RPMs on my Gentoo box just to get the PHP client working. Actually, I wouldn't mind if there was an ebuild.

Granted, I'm reading very little documentation while I do this, other than the PHP website, and going as far as this will take me intuitively. I've spent most of my time on Oracle, and their website is really nice, and has a section just for PHP. I still can't get any connections working though.

One thing's for sure, it makes me love PostgreSQL even more than ever. And I used to think it was a pain to setup. Holy crap, its a cakewalk compared to these guys.

I think it's time for some more cartoons. Green Lantern is definately the man. I think he's tougher than Superman.

I started working tonight on a SQL Server release of the lds-scriptures package. This one is actually going to have foreign keys, and maybe some views -- some of the big things that will be in the next point release for the other databases too.

Someday I'll have to get an eval version of Oracle and DB2 and maybe port it to those just for fun.

One of the main scripts was running slow tonight that gets hit on a regular basis, and we couldn't figure out why.

So, we do what we do when we aren't sure what to do -- we reindex the table. That's the first general fix we like to do. Turns out, that fixed it in this case, as well.

What had changed recently you might ask (and well so)? We added a column to one of the database tables.

I guess that was too much of a strain for SQL Server.

In today's adventure our hero ran into a small problem -- he wanted to select year-month-day from the database in the format of (2006-01-17). Easy, you would think, until you've actually worked with the database first hand.

First, I tried selecting them all individuall and concatenating them together .. which worked, except that datepart() won't pad the zeroes (it would return 2006-1-17).

Next, I decided to write my own UDF (User Defined Function for the unwashed masses), which would set the variables, but I ran into another problem -- you can't run getdate() within a UDF. That's weird (translation: That's about the dumbest thing I've seen this databse do yet).

Actually, the last problem turned out for a small bonus, because with my new UDF I could just pass it in as a variable anyway. Sure it complicates the function a little bit, but it will work out in the end.

Speaking of which, here is the final SQL for the function:

CREATE FUNCTION dateYMD(@getdate DATETIME) RETURNS VARCHAR(255) AS
BEGIN
DECLARE @year CHAR(4);
DECLARE @month VARCHAR(2);
DECLARE @day VARCHAR(2);
DECLARE @date VARCHAR(255);

SET @year = YEAR(@getdate);
SET @month = MONTH(@getdate);
SET @day = DAY(@getdate);

IF LEN(@day) = 1 BEGIN SET @day = '0' + @day END
IF LEN(@month) = 1 BEGIN SET @month = '0' + @month END

SET @date = @year + '-' + @month + '-' + @day;

RETURN @date;

END

Then, the final blow. After about getting halfway through writing this one, I find that there's an undcoumented feature (I guess those Microsoft developers enjoy Easter Eggs just as much as I do) to do the same thing I was doing:

SELECT CONVERT(VARCHAR, GETDATE(), 23);

SQL Server: 1, Developer: 0