moving 120gb of data between servers
NOTE: this is a re-post of my original entry from my old blog here
as i mentioned previously the transfer of 120+ gb did not go smoothly. we ended up tar’ing the directory into 1 gb chunks for transfer, that way if the connection timed out we could spot exactly what file was last successfully transfered. the tar command does this nicely, however, i didn’t do it so i can’t relay how. look it up i guess.
anyway so we got the files moved, another tar command re-assembled them, and hooray! duplicated 120 gb of data cloned. next is the database…
3 databases (mysql), ~60 tables total, with 6 tables having 500,000 + rows. lots of rows.
rule 1: haste != good. read things first. when phpmyadmin offers you an export option and gives you a “maximum rows” field, make sure that value is larger than what you’re trying to export. more importantly, don’t get confused/mad/frustrated when it doesn’t work because you went through the motions and didn’t read the instructions.
rule 2: when phpmyadmin gives you an import option that is limited to 2mb files, it’s limited to 2mb files. can you change this value somewhere, in some config file? probably, i don’t know. faster than you can load the data manually with some ssh and unix skills? probably not. transfer the .sql’s to the remote server that has the database on it and type this into the shell:
mysql -u username -p table_name < sqlfile.sql
make sure username has sufficient privileges. it helps.
rule 3: documentation is your friend i know it’s a pain to read sometimes, especially when you’re in the office on a saturday, and it’s barbecue weather outside, and coffee has replaced food as fuel for your body, but it’s there for a reason. a problem arrived with php’s setup: on the old server it was running under CGI, on new server it’s embedded into apache. the web app we were running required different permissions on different files depending on the case. the difference stems from permissions: when php is embedded things are run under the “apache user”, so permissions need to be set appropriately for it to changes things. specifically folders needed to be 755, contents needed to be 644. the web app, over the course of time, has created 7,000 + folders… lots to chmod on.
fortunately Find is an excellent helper in getting this done. basically, you find things that fit criteria (imagine that!) and tell it do something with it.
Find . -type d
this should find all things of type ‘d’ (d for directory) in location ‘.’, which in unix means “right here”. so make sure you’re where you want to be, say, the top directory of what needs to be changed. find then has an “-exec” option, so you can tell it to do something with all the stuff you find
Find . -type d -exec chmod 755 {}\;
this should change every folder in the current directory (recursively) and change the permissions to 755. i can’t say what “{}\;” really does. i know chmod wants another argument (chmod 777 thing), so i guess {} means “everything” or something like that.
so to change all the contents to 644
Find . -type f -exec chmod 644 {}\;
“f” for “regular file”
so there you go, that was the fun unix / mysql stuff i did over the weekend. we repeated the process today, as today is the cutoff, and the whole process took 1/5 the time once i knew what i was doing UPDATE:
Find . -type f -exec chmod 644 '{}' \;
find is lower case. i just looked up my own blog post to use this command and it wouldn't work. also, i added quotes around the brackets and added a space between closing bracket and the \;