Last visit was: It is currently Thu Dec 05, 2024 10:28 am


All times are UTC-05:00




Post new topic Reply to topic  [2 posts ] 
    Author Message
     Post subject:Memory efficiency on updating a dataset
    PostPosted:Sun May 31, 2009 9:24 am 
    External Project Staff
    User avatar
     

    Joined:Sun Oct 30, 2005 3:40 pm
    Posts:371
    Website:http://idlesoft.net
    Location:~/
    Case
    I have a large (newline separated) dataset which needs to be synchronized every day. The synchronization data comes in at a random order. Only data not already in the set should be added. Data in the set doesn't have to be updated. I have no control whatsoever over the input data.

    Problem
    I currently update this data using a Ruby Set. I load the current data into the set, load the external source, update the set, and write back. Only problem is this scripts memory usage has grown over time, currently using over 71mb on a very memory restricted system (a so called VPS ;)), causing the system to swap important data.

    Question
    How could I update this data far more memory efficient without slowing it down drastically?

    _________________
    -- ChaosR

    Image


    Top
    Offline  
     Post subject:Re: Memory efficiency on updating a dataset
    PostPosted:Tue Jun 09, 2009 6:55 am 
    Connoisseur of the Godawful
    User avatar
     

    Joined:Tue Mar 01, 2005 9:00 am
    Posts:456
    ICQ:286315965
    Website:http://rabidtinker.mine.nu/
    Yahoo Messenger:alistair_lynn
    AOL:Agent_Vast@mac.com
    Location:127.0.0.1
    Ruby isn't known for being compact, nor fast :)

    _________________
    Alastair Lynn / Alumnus / Onlink Team


    Top
    Offline  
    Display posts from previous: Sort by 
    Post new topic Reply to topic

      All times are UTC-05:00


      Who is online

      Users browsing this forum: No registered users and 1 guest


      You cannot post new topics in this forum
      You cannot reply to topics in this forum
      You cannot edit your posts in this forum
      You cannot delete your posts in this forum
      You cannot post attachments in this forum

      Search for:
      Jump to:  
      cron
      Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
      Theme created by Miah with assistance from hyprnova