Tim Booher

First I had to backup my database from Heroku — so I created the database in PostGresSQL.

heroku pgbackups:capture
curl -o latest.dump `heroku pgbackups:url`
pg_restore --verbose --clean --no-acl --no-owner -h localhost -d booher_blog latest.dump

Armed with that and the handy Induction I then needed to get my data into a format easily imported by WordPress. There is probably a standard way to do this, but I didn’t have time to find the efficient way. The first place I did look was the wordpress import data page, which led me to the right plugins. Then I was planning on running a command like:

COPY products_273 TO '/tmp/products_199.csv' DELIMITER ',' CSV HEADER;

It turns out that Induction has a great export tool to easily get me a new CSV file, but I had to make a new join table:

CREATE TABLE for_wp2 AS SELECT pages.title, page_parts.content, page_parts.name, pages.updated_at FROM pages, page_parts WHERE page_parts.page_id = pages.id;

Then I imported this via the wordpress-importer plugin. I used radiant “type” as a category and am going for the quick and dirty solution and figured I can manually copy over assets as opposed to creating a wget script to upload them. Wish there were an automated tool for this, but a little database knowledge is all that’s required.

So I deleted all my pictures, and I restored them which resulted in a bunch of corrupted images; thousands of corrupted images. To fix this, I wrote the following script in MATLAB using the image processing toolbox:

// insert blog here

Using matlab I tried to determine what a corrupted image is. First when using the image processing toolbox to open an image, I noticed:

g = imread(s);
Warning: JPEG library error (8 bit), "Corrupt JPEG data: premature end of data segment"."
Warning: JPEG library error (8 bit), "Invalid JPEG file structure: two SOI markers"."

Also, a histogram of such a file looked liked this:

So, the only challenge is to find the spike, or simply a crazy high percent of 128, the mean value. Simple enough.

cd('/media/95543211-fd8f-4fc9-9b24-3a787113e4c2/+JPEG');
jpegs = dir('.');

num_files = 100;

file_count = length(jpegs);

G = zeros(1,num_files-2);

for i = 3:(num_files+2)
    name = jpegs(i).name;
    disp(['working: ' name]);
    if true
        try
         I = rgb2gray(imread(name));
         [w, l] = size(I);
         gray_percent = sum(sum(I==128))/(w*l);
         G(i-2) = gray_percent;
         if gray_percent > 0.07
           disp(['moving . . . ' name]);
           movefile(name, ['too_much_gray/' name]);
         else
           disp(['good: ' name]);
           movefile(name, ['noerr/' name]);
         end
        catch
         disp(['bad: ' name]);
        end

    end
end

Then a script to see which images might be corrupt:

And a ruby script to move the results (yeah — really inefficient, I know).

So files that might crash matlab are at least removed.

#!/bin/bash

for f in *
do
  # echo "Processing $f file..."
  # take action on each file. $f store current file name
  if ! identify "$f" &> /dev/null; then
     echo "$f"
  fi
done

and (yes, this is silly)

#!/home/bonhoeffer/.rvm/rubies/ruby-1.9.3-p286/bin/ruby
filez = <<EOF
__003999
__026328
__029322
__032335
__035823
__035842
__036090
__038688
__039670
__048554
__048561
__048634
19991215_22_43_43_033877
19991215_22_43_43_034820
19991215_22_43_43_049844
19991215_22_43_56_038011
19991215_22_44_16_010202
20070729_14_42_57_048540
EOF

puts filez.split(' ').size

filez.split(' ').each do |f|
	puts "mv #{f} matlab_bad/#{f}"
	`mv #{f} matlab_bad/#{f}`
end

I had a lot of data in MySql in a rails application, but I moved to Mongo on a new application and wanted to give the client a nice archive of excel files for each client. It has been a long time since I’ve worked with visual basic, but it came back quickly.

I was amazed how clunky the interface and I wrote some pretty redundant code, but this was my Saturday and I tried to get this out the door quickly. My basic workflow was to load an office trial version on a virtual machine, then install SQL server on windows and set up an ODBC connection string which I used to import all data into access. In access, I just wrote a bunch of sql strings and used access to automate excel via VBA.

https://gist.github.com/tbbooher/2386826

This error was pretty hard to solve. I emailed heroku support and they recommended:

“https://github.com/ddollar/heroku-accounts”:https://github.com/ddollar/heroku-accounts

bq. For git commands to work git needs to use the ssh key associated with that account. It’s probably choosing a key associated with a different account which is causing these permissions errors. You can use a ~/.ssh/config file to specify which key should be used. Check out this client plugin written by one of our engineers. It helps manage both Heroku credentials and ssh keys for easy switching between accounts. https://github.com/ddollar/heroku-accounts Please let me know if you have more questions.

Their comment did make my future work with multiple accounts easier — it didn’t solve my problem which was that my old fingerprint was still active, and causing trouble.

The solution for me was not to create and provide a new key to heroku. It was to go to the gnome passwords and keys folder and manually delete “My Personal Keys”.

See this comment from “launchpad”:https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/505278 for details.

bq. The culprit is gpg-keyring-daemon. It subverts the normal operation of ssh-agent, mostly just so that it can pop up a pretty box into which you can type the passphrase for an encrypted ssh key. And it paws through your .ssh directory, and automatically adds any keys it finds to your agent. And it won’t let you delete those keys. How do we hate this? Let’s not count the ways — life’s too short.

bq. The failure is compounded because newer ssh clients automatically try all the keys in your ssh-agent when connecting to a host. If there are too many, the server will reject the connection. And since gnome-keyring-daemon has decided for itself how many keys you want your ssh-agent to have, and has autoloaded them, AND WON’T LET YOU DELETE THEM, you’re toast.

bq. What you really want to do is to turn off gpg-keyring-daemon altogether. Go to System –> Preferences –> Startup Applications, and unselect the “SSH Key Agent (Gnome Keyring SSH Agent)” box — you’ll need to scroll down to find it.

bq. You’ll still get an ssh-agent, only now it will behave sanely: no keys autoloaded, you run ssh-add to add them, and if you want to delete keys, you can. Imagine that.

[mathjax]

Here we tackle what a Radar needs to accurately measure a target’s position, size and motion. (This is nothing new, just a placeholder for some of my notes.)

First of all, everything depends on $SNR$, and the parameter estimation happens in the main computer of a Radar. The theoretical rms error of a measurement is of the form:

where \(k\) is a constant between 0.5 and 1.

h3. Range

First, let’s tackle range. They key metric for range is bandwidth, the higher the bandwidth the better the resolution.

Estimation of the range of a target is based upon using A/D sampled measurements of the round trip time to and the target

For time delay measurements , such as range, the value of the constant depends on the shape of the radar pulse’s spectrum and the pulse’s rise time and for a rectangular pulse, whose width is, (T):

or equivalently,

However, pulses always move in trains or if $T_D$ is the dwell time:

from Barton and Ward.

So Range depends on \(SNR\), pulse shape and width, effective bandwidth, number of pulses that can be coherently integrated.

This is explained well by a diagram in the lincoln presentation:

h3. Angle

Now, let’s tackle angle. Angle accuracy depends on antenna size. Simple detection provides coarse location in angle by isolating a target’s location within beamwidth of antenna (and beamwidth is just \(\lambda / D \), where \(D\) is the aperture diameter and \(\lambda \) is the wavelength. But this is very coarse, 1° beam at 100 km extends across 1,745 meters. In order to provide more accurate angle estimation, you have to measure at different beam positions to improve accuracy. You can do this through sequential lobing, conical scan tracking, or monopulse angle estimation.

h4. Conical Scanning

Conical scanning is a technique to keep the beam pointed at the target to improve angle accuracy. It is based on the principle that the radar receiver will get maximum returned signal strength when the target is in the beam centre, so if the beam is pointed directly at the target, when the target moves it will move out of the beam center and the received signal strength will fall. Circuitry designed to monitor any fall off in received signal strength controls a servo motor that steers the aerial to follow the target motion.

Wikipedia has a nice drawing that explains this technique:

!https://upload.wikimedia.org/wikipedia/commons/3/32/Conical_scan.jpg!

h4. Sequential lobing

While conical scan tracking uses the phase of modulation to get the angle error and amplitude modulation to get the beam displacement, sequential lobing uses two slightly separated antenna elements to send the beam slightly to either side of the midline of the antenna, switching between the two to find which one gave the stronger return, thereby indicating which direction the antenna should be moved in order to point directly at the target.

Here, wikipedia provides another good diagram:

!https://upload.wikimedia.org/wikipedia/commons/b/b6/Monopulseprinzip2.gif!

h4. Monopulse

Monopulse radars are similar in general construction to conical scanning systems, but add one more feature. Instead of broadcasting the signal out of the antenna “as is”, they split the beam into parts and then send the two signals out of the antenna in slightly different directions. When the reflected signals are received they are amplified separately and compared to each other, indicating which direction has a stronger return, and thus the general direction of the target relative to the boresight. While monopulse, requires a separate receiver for each channel, it improves performance over conical scan and sequential lobing whose performance degrade with time varying radar returns.

All angle errors are summed up well by the following:

and

SNR, type of measurement technique, antenna illumination distribution, antenna size, and frequency.

h3. Doppler

Lastly, let’s look at Doppler accuracy. Doppler accuracy depends on coherent integration time \(\Delta t \):

Doppler frequency is calculated as:

To accurately measure Doppler, use two closely spaced frequency filters offset from the center frequency of the Doppler filter containing the detection

Doppler accuracy depends on SNR, pulse shape, integration time.

h3. Real world limitations

h4. Receiver noise

# Adds variance to estimates

h4. Radar calibration

# Poor calibration leads to poor estimation

h4. Amplitude fluctuations

# Small effect on monopulse and array solutions

h4. Angle noise (angle scintillations, or target glint)

# Complex target return biases angle estimate

h4. Multipath (low angle tracking)

# Reflection off earth’s surface combines with direct path return

# Can cause biases in angle estimates for all technique

h4. Good References

“Parameter Estimation”:http://www.ll.mit.edu/workshops/education/videocourses/introradar/lecture9/lecture.pdf

“Basic Fundamentals”:http://faculty.nps.edu/jenn/Seminars/RadarFundamentals.pdf

“Parameter Estimation and Tracking”:http://aess.cs.unh.edu/Radar%202010%20PDFs/Radar%202009%20A_15%20Parameter%20Estimation%20and%20Tracking%20Part%201.pdf