Ferrous Moon
http://www.ferrousmoon.com:80/forums/

Memory efficiency on updating a dataset
http://www.ferrousmoon.com:80/forums/viewtopic.php?f=45&t=1796
Page 1 of 1

Author:  ChaosR [Sun May 31, 2009 9:24 am ]
Post subject:  Memory efficiency on updating a dataset

Case
I have a large (newline separated) dataset which needs to be synchronized every day. The synchronization data comes in at a random order. Only data not already in the set should be added. Data in the set doesn't have to be updated. I have no control whatsoever over the input data.

Problem
I currently update this data using a Ruby Set. I load the current data into the set, load the external source, update the set, and write back. Only problem is this scripts memory usage has grown over time, currently using over 71mb on a very memory restricted system (a so called VPS ;)), causing the system to swap important data.

Question
How could I update this data far more memory efficient without slowing it down drastically?

Author:  prophile [Tue Jun 09, 2009 6:55 am ]
Post subject:  Re: Memory efficiency on updating a dataset

Ruby isn't known for being compact, nor fast :)

Page 1 of 1 All times are UTC-05:00
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/