Tim Booher

First I had to backup my database from Heroku — so I created the database in PostGresSQL.

heroku pgbackups:capture
curl -o latest.dump `heroku pgbackups:url`
pg_restore --verbose --clean --no-acl --no-owner -h localhost -d booher_blog latest.dump

Armed with that and the handy Induction I then needed to get my data into a format easily imported by WordPress. There is probably a standard way to do this, but I didn’t have time to find the efficient way. The first place I did look was the wordpress import data page, which led me to the right plugins. Then I was planning on running a command like:

COPY products_273 TO '/tmp/products_199.csv' DELIMITER ',' CSV HEADER;

It turns out that Induction has a great export tool to easily get me a new CSV file, but I had to make a new join table:

CREATE TABLE for_wp2 AS SELECT pages.title, page_parts.content, page_parts.name, pages.updated_at FROM pages, page_parts WHERE page_parts.page_id = pages.id;

Then I imported this via the wordpress-importer plugin. I used radiant “type” as a category and am going for the quick and dirty solution and figured I can manually copy over assets as opposed to creating a wget script to upload them. Wish there were an automated tool for this, but a little database knowledge is all that’s required.

So I deleted all my pictures, and I restored them which resulted in a bunch of corrupted images; thousands of corrupted images. To fix this, I wrote the following script in MATLAB using the image processing toolbox:

// insert blog here

Using matlab I tried to determine what a corrupted image is. First when using the image processing toolbox to open an image, I noticed:

g = imread(s);
Warning: JPEG library error (8 bit), "Corrupt JPEG data: premature end of data segment"."
Warning: JPEG library error (8 bit), "Invalid JPEG file structure: two SOI markers"."

Also, a histogram of such a file looked liked this:

So, the only challenge is to find the spike, or simply a crazy high percent of 128, the mean value. Simple enough.

cd('/media/95543211-fd8f-4fc9-9b24-3a787113e4c2/+JPEG');
jpegs = dir('.');

num_files = 100;

file_count = length(jpegs);

G = zeros(1,num_files-2);

for i = 3:(num_files+2)
    name = jpegs(i).name;
    disp(['working: ' name]);
    if true
        try
         I = rgb2gray(imread(name));
         [w, l] = size(I);
         gray_percent = sum(sum(I==128))/(w*l);
         G(i-2) = gray_percent;
         if gray_percent > 0.07
           disp(['moving . . . ' name]);
           movefile(name, ['too_much_gray/' name]);
         else
           disp(['good: ' name]);
           movefile(name, ['noerr/' name]);
         end
        catch
         disp(['bad: ' name]);
        end

    end
end

Then a script to see which images might be corrupt:

And a ruby script to move the results (yeah — really inefficient, I know).

So files that might crash matlab are at least removed.

#!/bin/bash

for f in *
do
  # echo "Processing $f file..."
  # take action on each file. $f store current file name
  if ! identify "$f" &> /dev/null; then
     echo "$f"
  fi
done

and (yes, this is silly)

#!/home/bonhoeffer/.rvm/rubies/ruby-1.9.3-p286/bin/ruby
filez = <<EOF
__003999
__026328
__029322
__032335
__035823
__035842
__036090
__038688
__039670
__048554
__048561
__048634
19991215_22_43_43_033877
19991215_22_43_43_034820
19991215_22_43_43_049844
19991215_22_43_56_038011
19991215_22_44_16_010202
20070729_14_42_57_048540
EOF

puts filez.split(' ').size

filez.split(' ').each do |f|
	puts "mv #{f} matlab_bad/#{f}"
	`mv #{f} matlab_bad/#{f}`
end

I had a bunch of bogus images in my directory and I was going through and manually deleting them. I wised up a little bit and thought I would write a quick script. While it ran, I thought I would share with you.

I had: 25760 images and probably about 15,000 of them were bunk. I could use identify from ImageMagik to get the data which contained the width and height of the image. All the bogus images were a specific height. By writing a script I was able to put them all into a directory, which I could then review for deletion.

Enjoy