Database records implementation

[AsLan^]

So... I spent the last couple of days mucking about and making a database that stores all its records in a single plist file.

I decided to use a large data set (~88,000 records) and the plist file came out to ~13mb. So I was thinking, if I wrote the database to disk every time a record was modified, that would be a lot of writing. But if I waited until the app closed or the user specified to write to disk, then the modifications would be vulnerable to loss in the case of power failure, crash, etc.

So then, I got the idea of using a separate plist file for each record and committing after any particular record has been modified. This to me sounds like a great way to go, except in the case of high usage. In which case I expect The repeated disk accesses would not only slow down the program, but cause undue wear on the disk.

Any opinions on a strategy for implementing database records?

[Banana]

Forgive me if I'm being dense, but why should the plist size matter at all? Have your users access only a particular section of the list (let's say only records from a given period of time or in a certain section or whatever), for navigating. The database shouldn't be returning the whole thing every time a user access the database.

Look into filtering for more details if that is relevant.

HTH.

[AsLan^]

Forgiven! ;)

The way I've got it right now, the whole plist is loaded into memory when the program starts and bound to an NSTableView for navigation.

After it's all loaded, accessing any particular record is instantaneous and only returns the accessed record.

Writing it back to disk though is what I'm concerned with. It's conceivable that I could seek to the appropriate position in the file and make the changes but then I wouldn't be able to use Cocoa's nifty "writeToFile:" methods that make short work of arrays. And as far as total disk accesses are concerned, this would be practically the same as implementing a separate file for each record.

I'm I worrying about nothing? I know virtual memory gets shifted around all the time, can anyone estimate how many records per second would have to be accessed for the drive to be considered thrashing?

[euain]

It sounds like you should start looking at a database product of some kind? Seems exactly what they do. I am not familiar with Obj-C, but certainly in Javaland, there would be many free and simple candidates for this kind of (what sounds like straightforward) database application.

Alternatively:

Could you stick with a big file but write some kind of intent log. So while running you have big file (static) plus one or more (depending on how you implement it) files saying - add this record, remove this. At shutdown, you write to the main datafile and remove the small files. If it crashes, you lose power etc., you can use the small files to update the big data file.

Euain

[AsLan^]

Could you stick with a big file but write some kind of intent log. So while running you have big file (static) plus one or more (depending on how you implement it) files saying - add this record, remove this. At shutdown, you write to the main datafile and remove the small files. If it crashes, you lose power etc., you can use the small files to update the big data file.

Euain

That's not a bad idea actually...

It has the robustness of the individual files but eliminates the overhead involved with loading up all those files at start time and the complications of coming up with a system of uniquely naming files :)

As far as a database product is concerned, this is really just a personal project that I'm exploring :D

[Brad]

For what it's worth, unless this project is strictly an academic one, I would greatly recommend using an existing package like SQLite (http://developer.apple.com/documentation/MacOSX/Conceptual/OSX_Technology_Overview/AppTechnology/chapter_5_section_20.html) over implementing a brand new database in numerous plists, but I'm guessing this confirms the former?
As far as a database product is concerned, this is really just a personal project that I'm exploring :D

[bassplayinMacFiend]

Yea, if the program is only for you or a few people using Macs, why not build it as a CoreData project? If it's a simple database you probably wouldn't need much self-generated code.

[Mr Beardsley]

Yea, if the program is only for you or a few people using Macs, why not build it as a CoreData project? If it's a simple database you probably wouldn't need much self-generated code.

Totally agree. CoreData is incredible! You can get a database application going with 0 lines of code. Now granted it will be a little rough around the edges, but it will work. Here's a video that might help you get started with CoreData.

CoreData Video Tutorial (http://www.rentzsch.com/links/adcCoreDataVideoTutorial)

[AsLan^]

I"m not sure (still waiting for the video to download) what a CoreData app is exactly but I think that's close to what I might have done.

Basically, ala the Hillegass book I've gone and set the object of an NSArrayController to the record type and bound it to an NSTableView.

Where it departs from the book is that I've added form like interfaces for creating and editing the records, and saves to a plist.

I really like the way you can serialize an objects data into an NSDictionary, write it to disk, then read it back again with a real minimum of fuss. The records each consisted of three NSStrings, and it only takes about 30 seconds or so to read in and instantiate all 80,000+ records! (actually I have no idea if that's good or not but it seemed fairly impressive to me, perhaps it really is slow).

The idea I have is for a program that provides database functionality with a customized form interface and stores the data in a plain text easily manipulatable format (XML).

The reason being is that, a small business might not have the resources to hire a database manager (or a suitably specialized contractor to fix it when it goes wrong) and is considering using something like Access for their record keeping needs.

Of course, if they use Access, they get locked in.

The idea behind it being stored in plists is so that when the business grows to the point where they can think about hiring on full time IT staff, and need a higher performance solution, the data will all be stored in the very easily manipulatable plists.

Anyway... it really is just a proof of concept, I don't have the time to follow through unless someone agrees with my logic and pays me to do it :P

The thing that got me thinking about it was that one of my friends will be starting up a business soon and needed some advice about what to do for record keeping. He actually asked another friend, who asked me for advice, and that got me thinking of this kind of stepping stone database program, so I decided to make a quick proof of concept.

PS@Brad, I'll definitely take a look at that SQLite, I didn't realize it was just a library and didn't require any kind of DB finagling like MySQL or other such thing :)

[bassplayinMacFiend]

You can save CoreData as an XML document, IIRC. I know CD offers both a text and binary format. Just a thought. :)

[AsLan^]

The tutorial is still downloading! :lol:

[Mr Beardsley]

If your dataset really is 88,000 objects long, I would consider using CoreData with the SQLite data store. This is exactly what SQLite is designed for, and should be much faster than XML. Also, if you are worried about keeping data accessible in the future, you might look at using your plist approach to dump the database to XML. Use CoreData and SQLite as your day to day operational method (it will be much nicer and faster), but then provide an action that allows the user to dump the database to XML.

[AsLan^]

Well, I must say, CoreData is really, really, great! :D

I know what it was, and after watching the tutorial I still wasn't sure why it was so much better than creating records programmatically.

But I decided to redo the program as a CoreData app, and within an hour I was able to duplicate and exceed the functionality of my original program :D

Cheers everyone, and thank you for the tutorial link Mr. Beardsley!