Tim Booher

So I deleted all my pictures, and I restored them which resulted in a bunch of corrupted images; thousands of corrupted images. To fix this, I wrote the following script in MATLAB using the image processing toolbox:

// insert blog here

Using matlab I tried to determine what a corrupted image is. First when using the image processing toolbox to open an image, I noticed:

g = imread(s);
Warning: JPEG library error (8 bit), "Corrupt JPEG data: premature end of data segment"."
Warning: JPEG library error (8 bit), "Invalid JPEG file structure: two SOI markers"."

Also, a histogram of such a file looked liked this:

So, the only challenge is to find the spike, or simply a crazy high percent of 128, the mean value. Simple enough.

cd('/media/95543211-fd8f-4fc9-9b24-3a787113e4c2/+JPEG');
jpegs = dir('.');

num_files = 100;

file_count = length(jpegs);

G = zeros(1,num_files-2);

for i = 3:(num_files+2)
    name = jpegs(i).name;
    disp(['working: ' name]);
    if true
        try
         I = rgb2gray(imread(name));
         [w, l] = size(I);
         gray_percent = sum(sum(I==128))/(w*l);
         G(i-2) = gray_percent;
         if gray_percent > 0.07
           disp(['moving . . . ' name]);
           movefile(name, ['too_much_gray/' name]);
         else
           disp(['good: ' name]);
           movefile(name, ['noerr/' name]);
         end
        catch
         disp(['bad: ' name]);
        end

    end
end

Then a script to see which images might be corrupt:

And a ruby script to move the results (yeah — really inefficient, I know).

So files that might crash matlab are at least removed.

#!/bin/bash

for f in *
do
  # echo "Processing $f file..."
  # take action on each file. $f store current file name
  if ! identify "$f" &> /dev/null; then
     echo "$f"
  fi
done

and (yes, this is silly)

#!/home/bonhoeffer/.rvm/rubies/ruby-1.9.3-p286/bin/ruby
filez = <<EOF
__003999
__026328
__029322
__032335
__035823
__035842
__036090
__038688
__039670
__048554
__048561
__048634
19991215_22_43_43_033877
19991215_22_43_43_034820
19991215_22_43_43_049844
19991215_22_43_56_038011
19991215_22_44_16_010202
20070729_14_42_57_048540
EOF

puts filez.split(' ').size

filez.split(' ').each do |f|
	puts "mv #{f} matlab_bad/#{f}"
	`mv #{f} matlab_bad/#{f}`
end

I had a bunch of bogus images in my directory and I was going through and manually deleting them. I wised up a little bit and thought I would write a quick script. While it ran, I thought I would share with you.

I had: 25760 images and probably about 15,000 of them were bunk. I could use identify from ImageMagik to get the data which contained the width and height of the image. All the bogus images were a specific height. By writing a script I was able to put them all into a directory, which I could then review for deletion.

Enjoy

I had a lot of data in MySql in a rails application, but I moved to Mongo on a new application and wanted to give the client a nice archive of excel files for each client. It has been a long time since I’ve worked with visual basic, but it came back quickly.

I was amazed how clunky the interface and I wrote some pretty redundant code, but this was my Saturday and I tried to get this out the door quickly. My basic workflow was to load an office trial version on a virtual machine, then install SQL server on windows and set up an ODBC connection string which I used to import all data into access. In access, I just wrote a bunch of sql strings and used access to automate excel via VBA.

https://gist.github.com/tbbooher/2386826

I’m trying to get congressional district data to display on d3.js so I can do some cool overlays. The trick is to get matlab to comply.

This error was pretty hard to solve. I emailed heroku support and they recommended:

“https://github.com/ddollar/heroku-accounts”:https://github.com/ddollar/heroku-accounts

bq. For git commands to work git needs to use the ssh key associated with that account. It’s probably choosing a key associated with a different account which is causing these permissions errors. You can use a ~/.ssh/config file to specify which key should be used. Check out this client plugin written by one of our engineers. It helps manage both Heroku credentials and ssh keys for easy switching between accounts. https://github.com/ddollar/heroku-accounts Please let me know if you have more questions.

Their comment did make my future work with multiple accounts easier — it didn’t solve my problem which was that my old fingerprint was still active, and causing trouble.

The solution for me was not to create and provide a new key to heroku. It was to go to the gnome passwords and keys folder and manually delete “My Personal Keys”.

See this comment from “launchpad”:https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/505278 for details.

bq. The culprit is gpg-keyring-daemon. It subverts the normal operation of ssh-agent, mostly just so that it can pop up a pretty box into which you can type the passphrase for an encrypted ssh key. And it paws through your .ssh directory, and automatically adds any keys it finds to your agent. And it won’t let you delete those keys. How do we hate this? Let’s not count the ways — life’s too short.

bq. The failure is compounded because newer ssh clients automatically try all the keys in your ssh-agent when connecting to a host. If there are too many, the server will reject the connection. And since gnome-keyring-daemon has decided for itself how many keys you want your ssh-agent to have, and has autoloaded them, AND WON’T LET YOU DELETE THEM, you’re toast.

bq. What you really want to do is to turn off gpg-keyring-daemon altogether. Go to System –> Preferences –> Startup Applications, and unselect the “SSH Key Agent (Gnome Keyring SSH Agent)” box — you’ll need to scroll down to find it.

bq. You’ll still get an ssh-agent, only now it will behave sanely: no keys autoloaded, you run ssh-add to add them, and if you want to delete keys, you can. Imagine that.