Last visit was: It is currently Thu Mar 28, 2024 5:01 am


All times are UTC-05:00




Post new topic Reply to topic  [11 posts ] 
    Author Message
     Post subject:Large Project, need some help and tips
    PostPosted:Tue Jul 15, 2008 2:07 pm 
    User avatar
     

    Joined:Mon Mar 07, 2005 9:32 am
    Posts:600
    Location:localhost
    I've been working on a large reverse-lookup hash database (like the one at crysm) for several years now but want to take it commercial. I have some hardware and software model designs, but would like some feedback and ideas.

    I'm planning on building a small processing farm with a bunch of standard PC parts. The database itself is going to run on a machine with 8GB of PC2-6400, an Intel Q6600 and 10TB worth of RAID'ed 1TB hard drives. The five processing machines will have 2GB of PC2-6400 with an Intel Q6600. They will communicate through a 1Gb LAN switch. The server machine will issue jobs to the processing machines. When the job is complete, the resultant data will be sent back to the server machine where it is stored on a solid state (NAND flash) drive. The data will then be loaded into the database using the LOAD DATA INFILE command. For performance on hash lookups, the database will be split into 256 tables (prefix 00 through to FF) and hashes will be stored in hex with their first two digits removed (you don't need the full hash if you know the table prefix). Each hash algorithm will be given its own schema. The database management software will probably be Oracle on Windows Server 64-bit (to handle the 8GB of RAM). I considered Vista x64, but then remembered what a memory hog it was.

    Each job is issued as a starting point and a length, for example "Start at [2,1,27,8,13,1] length 100,000". The numbers are indexes of characters in a list (like a-z, A-Z, 0-9). The application hashes the plain text with many algorithms (CRC32, MD4, MD5, SHA variants, Whirlpool, RIPEMD160, etc), then stores them in a temporary database. Once the job is complete, the data is sent back to the server.

    I'm basically looking for any optimisation ideas regarding programming, database setup, data storage, hardware setup, system setup, etc.

    Oh, and if anyone fancies giving a helping hand with the financial side of things, feel free to PayPal me some cash :P

    _________________
    BAZINGA.


    Top
    Offline  
     Post subject:Re: Large Project, need some help and tips
    PostPosted:Tue Jul 15, 2008 5:16 pm 
    Organ Donor
    User avatar
     

    Joined:Mon Aug 13, 2007 1:47 pm
    Posts:529
    Location:Jawjuh
    Uh, isn't this sort of morally repulsive?

    _________________
    Creative people must be stopped! (Latest Entry 7/31/11: "Fishsticks (18+))

    Pleasantville by Night, a humorous horror web RPG


    Top
    Offline  
     Post subject:Re: Large Project, need some help and tips
    PostPosted:Tue Jul 15, 2008 9:56 pm 
    Literally Nine
    User avatar
     

    Joined:Tue Mar 01, 2005 9:00 am
    Posts:1263
    Um. Why?

    (Note: not a reply to Rickton)


    Top
    Offline  
     Post subject:Re: Large Project, need some help and tips
    PostPosted:Wed Jul 16, 2008 3:50 am 
    User avatar
     

    Joined:Mon Mar 07, 2005 9:32 am
    Posts:600
    Location:localhost
    Rickton:
    It's morally neutral. It's designed to be used for password recovery and network security. In fact one of the features I'm including is a password security check system - if the password can be found in my database, it's not safe to use. The system can be implemented into any company network or website sign-in page (for a fee of course). Yes, a few people out there are going to use it for bad, but that's the same with any security-based services. Of course, everyone will have to accept a disclaimer.

    Miah:
    It's a profitable business and there's a gap in the market. I'll set up a public trial where each IP address can search up to 5 hashes per day. After a week or so of the free trial, I'll switch the service back to commercial. Everyone is required to sign up with a credit card number. They get one or two free successful hash lookups to trial it. After that they only pay per month if they use the service and it gives them results. The payment bands are split up into a maximum number of successful queries per month, and if a client attempts to exceed that number, they are asked if they want to upgrade their account.

    _________________
    BAZINGA.


    Top
    Offline  
     Post subject:Re: Large Project, need some help and tips
    PostPosted:Thu Jul 17, 2008 3:30 pm 
    Connoisseur of the Godawful
    User avatar
     

    Joined:Tue Mar 01, 2005 9:00 am
    Posts:456
    ICQ:286315965
    Website:http://rabidtinker.mine.nu/
    Yahoo Messenger:alistair_lynn
    AOL:Agent_Vast@mac.com
    Location:127.0.0.1
    I would go for a distributed system. Anyone is open to run a 'server' daemon. Each server has a list of other servers, stored by some kind of unique ID and a hostname (probably a UUID). If a plaintext is requested, it'll check its own database, and if it finds nothing, delegate out to another server, which will itself follow the same procedure, returning a "chain" of UUIDs of servers from which the plaintext was fetched. Each one along the line has a random chance of checking the plaintext against its hash, so that it can blacklist UUIDs and inform its other hosts that one UUID is blacklisted, and if those other hosts receive more than a fixed number (say 3) of notices that one UUID needs to be blacklisted, it adds it to the blacklist. Any chain which has a blacklisted UUID in will be rejected. Now each server along the chain will add plaintexts from the hashes when it's fetched them, and each one is stored with a timestamp of when it was last used, so that older ones can be purged when the size of the DB reaches a fixed upper limit. Furthermore, each server has a 'boredom' threshold - if the load is under this threshold, it will pick a random method of finding plaintexts (such as picking a date and going through birthdays, picking words out of the dictionary, bruteforce methods with a random number of characters and a random characterset etc) and will add them to its local database. This way you distribute out the database among many hosts. Common plaintexts will require shorter chains but with enough servers all networked together, eventually a plaintext for any hash is likely to be found. Clients can just pick any host, and access them via the same TCP protocol.

    _________________
    Alastair Lynn / Alumnus / Onlink Team


    Top
    Offline  
     Post subject:Re: Large Project, need some help and tips
    PostPosted:Fri Jul 18, 2008 3:56 am 
    User avatar
     

    Joined:Mon Mar 07, 2005 9:32 am
    Posts:600
    Location:localhost
    I did consider distributed computing, but the problem is that with this kind of target market is that it will attract those who seek to damage it. It doesn't take much to break the hashing process on a client application (a single byte altered in the binary is probably enough), and in a commercial environment a poisoned hash is unacceptable. Also, making a distributed system work properly without serious development time (years) is practically impossible. The protocols need designing and a client discovery system (like a tracker) needs to be created. Furthermore, I already have a downscaled system working.

    Just a note on your "random check" idea - would it not just be better for the software to check the hash/plaintext match when they are searched for? That way 100% of fake hashes are caught. The random checks and blacklists would be useful though in conjunction with this, as it would block out clients that persistently send bad hashes.

    _________________
    BAZINGA.


    Top
    Offline  
     Post subject:Re: Large Project, need some help and tips
    PostPosted:Fri Jul 18, 2008 10:55 am 
    Connoisseur of the Godawful
    User avatar
     

    Joined:Tue Mar 01, 2005 9:00 am
    Posts:456
    ICQ:286315965
    Website:http://rabidtinker.mine.nu/
    Yahoo Messenger:alistair_lynn
    AOL:Agent_Vast@mac.com
    Location:127.0.0.1
    Well, my thinking was saving processing time. You could just set the random chance threshold to 1 so that they're all checked.

    _________________
    Alastair Lynn / Alumnus / Onlink Team


    Top
    Offline  
     Post subject:Re: Large Project, need some help and tips
    PostPosted:Sat Jul 19, 2008 3:24 am 
    User avatar
     

    Joined:Mon Mar 07, 2005 9:32 am
    Posts:600
    Location:localhost
    All I'm looking for is tips for efficiency regarding my current system, not a new system.

    _________________
    BAZINGA.


    Top
    Offline  
     Post subject:Re: Large Project, need some help and tips
    PostPosted:Sat Jul 19, 2008 12:45 pm 
    Connoisseur of the Godawful
    User avatar
     

    Joined:Tue Mar 01, 2005 9:00 am
    Posts:456
    ICQ:286315965
    Website:http://rabidtinker.mine.nu/
    Yahoo Messenger:alistair_lynn
    AOL:Agent_Vast@mac.com
    Location:127.0.0.1
    Make sure you check every query for places you can add indices.

    _________________
    Alastair Lynn / Alumnus / Onlink Team


    Top
    Offline  
     Post subject:Re: Large Project, need some help and tips
    PostPosted:Sat Jul 19, 2008 2:10 pm 
     

    Joined:Sun Jun 10, 2007 11:41 am
    Posts:344
    Location:Ninjaville
    I know very little about encryption but Godamn that sounds like a system I would like to own.


    Top
    Offline  
     Post subject:Re: Large Project, need some help and tips
    PostPosted:Sun Jul 20, 2008 10:41 pm 
    User avatar
     

    Joined:Sun Feb 12, 2006 8:56 pm
    Posts:1019
    Website:http://eddieringle.com
    Location:Detroit, MI
    Quote:
    I know very little about encryption but Godamn that sounds like a system I would like to own.
    Seconded.

    _________________
    -- Eddie Ringle

    Check out Elysian Shadows and consider backing us on Kickstarter!

    ====================================

    Image


    Top
    Offline  
    Display posts from previous: Sort by 
    Post new topic Reply to topic

      All times are UTC-05:00


      Who is online

      Users browsing this forum: No registered users and 12 guests


      You cannot post new topics in this forum
      You cannot reply to topics in this forum
      You cannot edit your posts in this forum
      You cannot delete your posts in this forum
      You cannot post attachments in this forum

      Search for:
      Jump to:  
      cron
      Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
      Theme created by Miah with assistance from hyprnova