Not a tame lion...
Join Date: May 2004
Location: Narnia
|
So... I spent the last couple of days mucking about and making a database that stores all its records in a single plist file.
I decided to use a large data set (~88,000 records) and the plist file came out to ~13mb. So I was thinking, if I wrote the database to disk every time a record was modified, that would be a lot of writing. But if I waited until the app closed or the user specified to write to disk, then the modifications would be vulnerable to loss in the case of power failure, crash, etc. So then, I got the idea of using a separate plist file for each record and committing after any particular record has been modified. This to me sounds like a great way to go, except in the case of high usage. In which case I expect The repeated disk accesses would not only slow down the program, but cause undue wear on the disk. Any opinions on a strategy for implementing database records? |
quote |
is the next Chiquita
Join Date: Feb 2005
|
Forgive me if I'm being dense, but why should the plist size matter at all? Have your users access only a particular section of the list (let's say only records from a given period of time or in a certain section or whatever), for navigating. The database shouldn't be returning the whole thing every time a user access the database.
Look into filtering for more details if that is relevant. HTH. |
quote |
Not a tame lion...
Join Date: May 2004
Location: Narnia
|
Forgiven!
The way I've got it right now, the whole plist is loaded into memory when the program starts and bound to an NSTableView for navigation. After it's all loaded, accessing any particular record is instantaneous and only returns the accessed record. Writing it back to disk though is what I'm concerned with. It's conceivable that I could seek to the appropriate position in the file and make the changes but then I wouldn't be able to use Cocoa's nifty "writeToFile:" methods that make short work of arrays. And as far as total disk accesses are concerned, this would be practically the same as implementing a separate file for each record. I'm I worrying about nothing? I know virtual memory gets shifted around all the time, can anyone estimate how many records per second would have to be accessed for the drive to be considered thrashing? |
quote |
Member
Join Date: Jan 2005
Location: UK
|
It sounds like you should start looking at a database product of some kind? Seems exactly what they do. I am not familiar with Obj-C, but certainly in Javaland, there would be many free and simple candidates for this kind of (what sounds like straightforward) database application.
Alternatively: Could you stick with a big file but write some kind of intent log. So while running you have big file (static) plus one or more (depending on how you implement it) files saying - add this record, remove this. At shutdown, you write to the main datafile and remove the small files. If it crashes, you lose power etc., you can use the small files to update the big data file. Euain |
quote |
Not a tame lion...
Join Date: May 2004
Location: Narnia
|
Quote:
It has the robustness of the individual files but eliminates the overhead involved with loading up all those files at start time and the complications of coming up with a system of uniquely naming files As far as a database product is concerned, this is really just a personal project that I'm exploring |
|
quote |
Selfish Heathen
Join Date: May 2004
Location: Zone of Pain
|
For what it's worth, unless this project is strictly an academic one, I would greatly recommend using an existing package like SQLite over implementing a brand new database in numerous plists, but I'm guessing this confirms the former?
The quality of this board depends on the quality of the posts. The only way to guarantee thoughtful, informative discussion is to write thoughtful, informative posts. AppleNova is not a real-time chat forum. You have time to compose messages and edit them before and after posting. |
quote |
Banging the Bottom End
Join Date: Jun 2004
|
Yea, if the program is only for you or a few people using Macs, why not build it as a CoreData project? If it's a simple database you probably wouldn't need much self-generated code.
|
quote |
Member
|
Quote:
CoreData Video Tutorial "Slow vehicle speeds with frequent stops would signal traffic congestion, for instance." uh... it could also signal that my Mom is at the wheel... |
|
quote |
Not a tame lion...
Join Date: May 2004
Location: Narnia
|
I"m not sure (still waiting for the video to download) what a CoreData app is exactly but I think that's close to what I might have done.
Basically, ala the Hillegass book I've gone and set the object of an NSArrayController to the record type and bound it to an NSTableView. Where it departs from the book is that I've added form like interfaces for creating and editing the records, and saves to a plist. I really like the way you can serialize an objects data into an NSDictionary, write it to disk, then read it back again with a real minimum of fuss. The records each consisted of three NSStrings, and it only takes about 30 seconds or so to read in and instantiate all 80,000+ records! (actually I have no idea if that's good or not but it seemed fairly impressive to me, perhaps it really is slow). The idea I have is for a program that provides database functionality with a customized form interface and stores the data in a plain text easily manipulatable format (XML). The reason being is that, a small business might not have the resources to hire a database manager (or a suitably specialized contractor to fix it when it goes wrong) and is considering using something like Access for their record keeping needs. Of course, if they use Access, they get locked in. The idea behind it being stored in plists is so that when the business grows to the point where they can think about hiring on full time IT staff, and need a higher performance solution, the data will all be stored in the very easily manipulatable plists. Anyway... it really is just a proof of concept, I don't have the time to follow through unless someone agrees with my logic and pays me to do it :P The thing that got me thinking about it was that one of my friends will be starting up a business soon and needed some advice about what to do for record keeping. He actually asked another friend, who asked me for advice, and that got me thinking of this kind of stepping stone database program, so I decided to make a quick proof of concept. PS@Brad, I'll definitely take a look at that SQLite, I didn't realize it was just a library and didn't require any kind of DB finagling like MySQL or other such thing |
quote |
Banging the Bottom End
Join Date: Jun 2004
|
You can save CoreData as an XML document, IIRC. I know CD offers both a text and binary format. Just a thought.
|
quote |
Not a tame lion...
Join Date: May 2004
Location: Narnia
|
The tutorial is still downloading!
|
quote |
Member
|
If your dataset really is 88,000 objects long, I would consider using CoreData with the SQLite data store. This is exactly what SQLite is designed for, and should be much faster than XML. Also, if you are worried about keeping data accessible in the future, you might look at using your plist approach to dump the database to XML. Use CoreData and SQLite as your day to day operational method (it will be much nicer and faster), but then provide an action that allows the user to dump the database to XML.
"Slow vehicle speeds with frequent stops would signal traffic congestion, for instance." uh... it could also signal that my Mom is at the wheel... |
quote |
Not a tame lion...
Join Date: May 2004
Location: Narnia
|
Well, I must say, CoreData is really, really, great!
I know what it was, and after watching the tutorial I still wasn't sure why it was so much better than creating records programmatically. But I decided to redo the program as a CoreData app, and within an hour I was able to duplicate and exceed the functionality of my original program Cheers everyone, and thank you for the tutorial link Mr. Beardsley! |
quote |
Posting Rules | Navigation |
|
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
I need a database software to keep records of people | usurp | Third-Party Products | 6 | 2006-10-03 12:48 |
Slow Database Performance | screensaver400 | Genius Bar | 0 | 2006-07-27 15:52 |
MySQL database disabled, how to "restart" it? | drewprops | Programmer's Nook | 3 | 2005-12-31 19:17 |
Tiger Mail junk filter database | Bill M | Genius Bar | 3 | 2005-12-27 14:46 |
Microsoft Access? | Messiahtosh | Third-Party Products | 26 | 2004-11-18 08:56 |