Last visit was: It is currently Tue Dec 11, 2018 9:55 pm


All times are UTC-05:00




Post new topic  Reply to topic  [ 2 posts ] 
Author Message
 Post subject: Memory efficiency on updating a dataset
PostPosted: Sun May 31, 2009 9:24 am 
External Project Staff
User avatar
 

Joined: Sun Oct 30, 2005 3:40 pm
Posts: 371
Website: http://idlesoft.net
Location: ~/
Case
I have a large (newline separated) dataset which needs to be synchronized every day. The synchronization data comes in at a random order. Only data not already in the set should be added. Data in the set doesn't have to be updated. I have no control whatsoever over the input data.

Problem
I currently update this data using a Ruby Set. I load the current data into the set, load the external source, update the set, and write back. Only problem is this scripts memory usage has grown over time, currently using over 71mb on a very memory restricted system (a so called VPS ;)), causing the system to swap important data.

Question
How could I update this data far more memory efficient without slowing it down drastically?

_________________
-- ChaosR

Image


Top
Offline   
 Post subject: Re: Memory efficiency on updating a dataset
PostPosted: Tue Jun 09, 2009 6:55 am 
Connoisseur of the Godawful
User avatar
 

Joined: Tue Mar 01, 2005 9:00 am
Posts: 456
ICQ: 286315965
Website: http://rabidtinker.mine.nu/
Yahoo Messenger: alistair_lynn
AOL: Agent_Vast@mac.com
Location: 127.0.0.1
Ruby isn't known for being compact, nor fast :)

_________________
Alastair Lynn / Alumnus / Onlink Team


Top
Offline   
Display posts from previous:  Sort by  
Post new topic  Reply to topic

All times are UTC-05:00


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
Theme created by Miah with assistance from hyprnova