Tim Booher

MathJax.Hub.Config({tex2jax: {inlineMath: [[‘$’,’$’], [‘(‘,’)’]]}});

I wanted to be able to do this with my ip address:

I do development and consulting in two areas I really love: fitness and political engagement. For some work I’m doing on the latter, I needed to figure out the latitude and longitude of the geographic center for each of the 437 US congressional districts in order to display the districts using the google maps API. “GovTrack”:http://www.govtrack.us/congress/findyourreps.xpd has done the hard work to get the data from the census to KML and as a WMS service so congressional districts can be displayed on Google maps, but there was no source online to get the geographic centers for these regions and google maps requires a zoom level and geographic center for the map.

The first question I had to face, was what a geographic centroid really is. Matlab has a “meanm”:http://www.mathworks.com/help/toolbox/map/ref/meanm.html function, which can find the center of a number of points on an arbitrary geodetic ellipsoid (e.g. WGS84), but I really felt the point should be inside the district. My first thought was to write a quick genetic algorithm that found the point that minimized the average distance to the exterior of the district, but it took forever to run and I was on ground that smelled as if smarter folks had trod here. This suspicion led me to look into some research done by the Royal Melbourne Institute of Technology which cataloged how to compute geographic centroids a number of different ways, the most relevant being the @moment@ @centroid@. This took a good bit of time to play with an implement, but I’ll refer the discussion to their paper. The end result was that *meanm* was good enough and the genetic algorithm was canned. (Email me if you want the code.)

From their paper:
The New Shorter Oxford English Dictionary (SOED 1993) defines the centroid as: “A point defined in relation to a given figure in a manner analogous to the centre of mass of a corresponding body.” Using this definition, and regarding the body as a plane area A of uniformly thin material, its centroid is
$$begin{matrix}
{bar y} = frac{M_y}{A} = frac{M_x}{A}
end{matrix}$$
are (first) moments with respect to the x- and y-axes respectively.
[The moment $M_L$ of a plane area with respect to a line L is the product of the area and the perpendicular distance of its centroid from the line.] The centroid computed using this method has a physical characteristic that is intuitively reassuring. That is, if we cut out a shape from uniformly thin material (say thin cardboard) and suspend it freely on a string connected to its centroid, the shape will lie horizontal in the earth’s gravity field.

I’m sharing a bit of my code, in the hope that some of the other folks in the OpenGovernment movement will find these datapoints useful. The first challenge was to find the (very poorly documented) “source data”:http://www.census.gov/geo/www/cob/cd110.html and get it into Matlab. The source code below is fairly straightforward: some data conditioning then find the geographic centers based on *meanm*. The moment centroid was interesting, but took too long to compute for my timeline, and the trival solution seems to be working well, the remaining challenge was to work with the data, something my day job has me practice a good bit.

One of the biggest challenges was to get the US census data into a standard two letter postal abbreviation format, the first part of my primary key (XXYY), where XX is the state and YY is the district number. The census had an arbitrary number assigned to each state without any documentation that I could find, so I had to resort to using regular expressions with ruby on the file names to pull out the data which would create a nice Matlab cell array, so I could then get a good ruby hash of the centroids from the Matlab script. Those two codes follow for any other data conditioners out there who might find them useful. I had to two challenges that required fancy regular expressions: to get the census number to US state, then to get the full state name to the two letter code.

And, in conclusion, I provide the geographic centers of all 437 congressional districts:

So this is really pretty boring, but now that I am working with this dataset, I hope to have some future posts on interesting GIS computations which combine demographics with members voting patterns. Please let me know if there are any questions or interesting ideas for exploration.

There are other blogs out there telling you how to set up a ruby
server install with ubuntu, nginx, passenger, linode ubuntu 10.10, and
rvm, but I couldn’t find one that put it all together, so I decided to
share my setup from beginning to end. I hope it helps you, but I’m also
hoping you point out any risks that I should clean up.

In general I used “this blog”:http://thekindofme.wordpress.com/2010/10/24/rails-3-on-ubuntu-10-10-with-rvm-passenger-and-nginx/, but it didn’t have everything I needed, as you’ll see below.

First I set up the server in linode, then connected to the server via a simple ssh command:

ssh root@tacitus

Before anything else, I had to install basic applications such as git and emacs.

sudo apt-get install git emacs

I decided that I wanted to set up everything from the perspective of
the deployment user, passenger, so I created the user *passenger*:

root@li289-94:~# useradd -d /home/passenger -m passenger
root@li289-94:~# usermod -s /bin/bash passenger
root@li289-94:~# passwd passenger

Now set-up ssh with public keys to enable password free auto-connection. Then I logged in as root, added passenger
to sudoers (using visudo and adding passenger to the sudo group) and started getting ready to install the ruby
version manager, _rvm_.

bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

Now, we need to load RVM into a shell session to .bashrc to make the rvm executables available.

added [[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm"

To prevent restarting the shell, you can run source ~/.rvm/scripts/rvm to get everything up and running immediately.
In order to test the installation, I tried ‘type rvm | head -n1’ to be sure rvm was a function available to the user.

Now, it’s time to get everything ready for the heavy lifting a ruby install requires.

sudo aptitude install build-essential bison openssl libreadline5
libreadline5-dev curl git-core zlib1g zlib1g-dev libssl-dev
libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libmysqlclient-dev

And I forgot to run (needed later):

sudo apt-get install libxslt-dev libxml2-dev libcurl4-openssl-dev

And the big moment, the chance to let rvm handle the latest ruby. (This will take a good bit of time.)

rvm install 1.9.2
rvm use 1.9.2 --default

With 1.9.2 installed, we can now configure passenger and create a gemset for our project

rvm 1.9.2 --passenger
rvm gemset create polco
rvm 1.9.2@polco
gem install passenger
rvmsudo passenger-install-nginx-module

Stick with the passenger defaults. The only problem was on rvm 1.9.2 –passenger, I get:

NOTE: If you are using Passenger 3 you no longer need the passenger_ruby,
use the wrapper script for your ruby instead (see 'rvm wrapper')

This was a new one to me, and made me deviate from the script I was following. I am using passenger-3.0.6, so I am definitely more up to date than the server. Due to this error, I broke with the blog I was following and did not change passenger_ruby to:

passenger_ruby /home/wayne/.rvm/bin/passenger_ruby;

This might still be something I need to do since I am currently getting a 404. I am changing back to this — which for me is the following line:

passenger_ruby /home/passenger/.rvm/bin/passenger_ruby;

h3. Passenger Configuration

gem install passenger

I chose, “compile and install nginx for me” and chose default install
locations. This takes a good while too. Now it was time to set up
nginx.

The installer told me that my configuration was already modified and
that:

This installer has already modified the configuration file for you! The
following configuration snippet was inserted:

  http {
      ...
      passenger_root /home/passenger/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.6;
      passenger_ruby /home/passenger/.rvm/wrappers/ruby-1.9.2-p180/ruby;
      ...
  }

To prepare for the deployment, I made the deployment directory:

mkdir -p /home/passenger/polco/public/;

So now I set up a basic server with the following nginx.conf file:

worker_processes  1;
 error_log /var/log/nginx/error.log debug;

 events {
    worker_connections  1024;
 }

 http {
    passenger_root /home/passenger/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.6;
#    passenger_ruby /home/passenger/.rvm/wrappers/ruby-1.9.2-p180/ruby;
    passenger_ruby /home/passenger/.rvm/bin/passenger_ruby;

    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;

    keepalive_timeout  65;

    server {
      listen 80;
      server_name localhost;
      root /home/deployer/polco/public;
      passenger_enabled on;
   }

 }

And I installed an nginx start script:

cd
git clone git://github.com/jnstq/rails-nginx-passenger-ubuntu.git
sudo mv rails-nginx-passenger-ubuntu/nginx/nginx /etc/init.d/nginx
sudo chown root:root /etc/init.d/nginx

Now I am making a single-user git repo:

mkdir ~/polco.bare.git/
cd polco.bare.git/
git init --bare

So I could now run from my desktop:

git push ssh://passenger@tacitus/~/polco.bare.git/ master

Everything worked . . . so far, now for the hard part . . .

h3. Deployment Setup

On the server, installed the bundler gem (under the current rvm: ruby 1.9.2@polco) and put together the following deploy.rb with Capistrano.

 $:.unshift(File.expand_path("/home/tim/.rvm/lib"))
 require 'rvm/capistrano'
 set :rvm_ruby_string, '1.9.2'
 set :rvm_type, :user
 require 'bundler/capistrano'
 set :application, "polco"
 role :web, "66.228.39.94"
 role :app, "66.228.39.94"
 role :db,  "66.228.39.94", :primary => true
 default_run_options[:pty] = true
 ssh_options[:forward_agent] = true
 set :deploy_to, "/home/passenger/polco"
 set :deploy_via, :remote_cache
 set :user, "passenger"
 set :use_sudo, false
 set :scm, :git
 set :scm_username, "passenger"
 set :repository, "ssh://tim@tacitus/~/polco.git/"
 set :branch, "master"
 set :git_enable_submodules, 1
 namespace :deploy do
  task :start, :roles => :app do
    run "touch #{current_path}/tmp/restart.txt"
  end

  task :stop, :roles => :app do

  end

  desc "Restart Application"
  task :restart, :roles => :app do
    run "touch #{current_path}/tmp/restart.txt"
  end

  desc "Symlink shared resources on each release - not used"
  task :symlink_shared, :roles => :app do

  end
 end

p. after ‘deploy:update_code’, ‘deploy:symlink_shared’

Then, from my desktop, I built the necessary directories and tested my deployment.

cap deploy:setup
cap deploy

The deployment works fine, and I know I still need to get my databases setup on the server for everything to run correctly. But I am surprised that nginx gives me a 404.

2011/04/11 22:52:00 [notice] 20057#0: signal 17 (SIGCHLD) received
 2011/04/11 22:52:00 [notice] 20057#0: worker process 20059 exited with code 0
 2011/04/11 22:52:00 [notice] 20057#0: exit
 2011/04/11 22:52:00 [notice] 20712#0: using the "epoll" event method
 2011/04/11 22:52:00 [notice] 20712#0: nginx/0.8.54
 2011/04/11 22:52:00 [notice] 20712#0: built by gcc 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu5)
 2011/04/11 22:52:00 [notice] 20712#0: OS: Linux 2.6.38-linode31
 2011/04/11 22:52:00 [notice] 20712#0: getrlimit(RLIMIT_NOFILE): 1024:1024
 2011/04/11 22:52:00 [notice] 20752#0: start worker processes
 2011/04/11 22:52:00 [notice] 20752#0: start worker process 20753
 2011/04/11 22:52:16 [error] 20753#0: *1 "/home/deployer/polco/public/index.html" is not found (2: No  such file or directory), client: 71.191.237.168, server: localhost, request: "GET / HTTP/1.1", host:  "tacitus"

Please let me know of any comments or ways I can improve my setup. Also, I would love to hear if any of this was helpful for you.

I like to automate Microsoft Visio to draw plots. They look so much more professional and I don’t have to fuss with Excel to figure out how to make plots look like I want. Again, this is me practicing my theory of being in control, not letting software control us. Please let me know if this code is useful.

This is my source code for the stacked bar plot I’m working on. Built as an excel Macro, it automates Visio to build the plot.

Sub BuildPlot()

Dim oApp As Excel.Application
Dim oWB As Excel.Workbook

Dim oDoc As Visio.Document
Dim oPage As Visio.Page
Dim oShape As Visio.Shape

Set oApp = New Excel.Application
Set oWB = oApp.Workbooks.Open("c:usersTimothy.BooherDesktopbar_plot_macro.xlsx")

Set oDoc = Application.ActiveDocument
Set oPage = Application.ActivePage

Dim iRow As Integer
Dim iAircraft As Integer
Dim x As Double
Dim x_prime As Double
Dim y As Double
Dim y_prime As Double
Dim xWidth As Double
Dim xGap As Double
Dim iStart As Integer

xWidth = 0.5
xGap = 2
x = 0
iStart = 2

For iAircraft = iStart To (iStart + 4)
    Debug.Print oWB.sheets(1).Cells(iAircraft, 2).Value
    x = x + xGap
    x_prime = x + xWidth
    y_prime = 0
    For iRow = 7 To 4 Step -1
        Debug.Print oWB.sheets(1).Cells(iAircraft, iRow).Value
        If oWB.sheets(1).Cells(iAircraft, iRow).Value > 0 Then
            y = y_prime
            y_prime = y + oWB.sheets(1).Cells(iAircraft, iRow).Value * 0.8
            Set oShape = oPage.DrawRectangle(x, y, x_prime, y_prime)
            oShape.Cells("FillForegnd") = 8 - iRow
        End If
    Next iRow
Next iAircraft

oWB.Close

Set oApp = Nothing

End Sub

Sub draw_axes()

Dim oDoc As Visio.Document
Dim oPage As Visio.Page
Dim oShape As Visio.Shape

Set oDoc = Application.ActiveDocument
Set oPage = Application.ActivePage

Dim i As Integer ' y
Dim j As Integer ' x

Dim numX  As Integer
Dim numY As Integer
Dim max_val As Double
Dim y_new As Double

numX = 10
numY = 10

max_val = 15 ' Ceiling(14.65)

' draw y axis
oPage.DrawLine 0, 0, 0, max_val

y_new = 0

For i = 1 To numY + 1
 'DrawLine
 oPage.DrawLine 0, y_new, 0.1, y_new
 y_new = y_new + max_val / numY
Next i

' draw x
'For j = 1 To numX
'
'Next j


End Sub

Sub build_conf_intervals()

Dim oDoc As Visio.Document
Dim oPage As Visio.Page
Dim oShape As Visio.Shape

Set oDoc = Application.ActiveDocument
Set oPage = Application.ActivePage

Dim iAircraft As Integer
Dim iBar As Integer

Dim y(1) As Double
delta_x = 3
to_wit = 0.1

x = 0

For iAircraft = 16 To 20
    x = x + delta_x
    For iBar = 2 To 3
        y(iBar - 2) = oWB.sheets(1).Cells(iAircraft, iBar).Value
    Next iBar
    oPage.DrawLine x, y(0), x, y(1)
    oPage.DrawLine x - to_wit, y(1), x, y(1)
    oPage.DrawLine x - to_wit, y(0), x, y(0)
Next iAircraft

oWB.sheets(1).Cells(iAircraft, 2).Value

End Sub

I have a bad, bad networking problem with my wife’s MacBook. She can’t connect to the internet anywhere, but I am working with my Netgear WNDR3300 for this problem.

Our wireless router is dd-wrt based (open source router firmware), and works fine for about 20 other nodes (other laptops, guests, mobile devices, etc). A couple of details were helpful to start with. Her MacBook has three interfaces: en0, en1, and fw0.

* en0 Ethernet Network Interface Controller (NIC)
* en1 WiFi Network Interface Controller (NIC)
* fw0 Firewire Networking

The second interface (en1) got an ip of 196.254.153.219 which is a link local address (anything in 169.254.0.0/16) is supposed to mean I didn’t get an ip from the DHCP server. However, I could see 192.168.1.1, which is the local ip for the router. This confused me, because it told me that the radio was working and I was getting some type of ip, ah I just don’t understand how I could get a partial connection to the router.

So I went into the Airport Utility, and got the message that AirPort Utility was unable to find any wireless devices. What! With a connection to the router? I was still confused. How was this possible. I was thinking about trying to reseat the wireless card, looking at buying a new wireless card, but that seemed like a lot of work that might not even solve the problem.

Edit, the Airport Utility is not for wireless interfaces, it is for Airport Express/AppleTV like items

I tried to ping google’s DNS server 8.8.8.8, with no host.

The first thing I did was try to manually set the IP to avoid a DHCP error, but the Airport utility said I had no devices.

I was able to ping the wan port (172.27.35.xxx), but I couldn’t ping the public ip (74.96.53.xxx) from verizon dsl. So I thought the issue was my modem, but the laptop was having connection problems, so I figured I had to reseat the wireless card. OSX said I had an AirPort extreme, Broadcom BCM43xx 1.0 (5.10.91.21). Hmm . . . looks connected, but maybe it’s loose? I’m out of options.

From the software, I found my wife has a MacBook5,2, which from Wikipedia is the MB881*/A with the Integrated 802.11a/b/g and draft-n (n enabled). I followed the guide here to confirm the card is in correctly: http://www.ifixit.com/Guide/Repair/Installing-MacBook-Core-2-Duo-AirPort-Card/519/ and everything checked out just fine. That took a lot of time, but confirmed that the card is not loose.

Then I went into network settings and set a manual ip. My subnet mask is 255.255.255.0 with a dhcp server serving from 192.168.1.100 with 50 maximum dhcp users. I am also using opendns for content filtering and speed advantages. So I went into network settings and assigned an IPv4 address of 192.168.1.80 with the correct subnet and router address. I next went to terminal and used ifconfig to see my ip on en1 was 192.168.1.80 and broadcast of 192.168.1.255, media was listed as autoselect status: active.

And . . . I got ping returns from 8.8.8.8 and 4.2.2.2 — name servers I used for testing. However, my default test mit.edu returned nothing (the same with www.google.com). I was able to load http://74.125.225.18 (the site for google).

O.K. so then i knew it was a DNS problem. I hit the terminal with dscacheutil -flushcache to flush the cache, and then restarted. I was worried in that I couldn’t find my NAS on my network, but I hoped a restart would cure all ills. It didn’t: still no www.mit.edu or www.yahoo.com. Hmm . . . Branes from #mac on irc recommended that I manually add the DN servers from OpenDNS. I added the standard 208.67.222.222 and 208.67.220.220 in the advanced mac en1 settings . . . and . . . working great.

Still don’t know why I was getting a local link address. Oh well, it is working just fine now. Any thoughts appreciated.

How I deal with these will be the subject of a future post. Please let me know any pointers on any of this.

“

So you didn’t set up automatic backups and need to recover your data? Don’t try digging it out of mysql, look no further than your rails production log. OK, try a bit with mysql, because the rails production log is a terrible way to recover your data. Only use as a last resort, but when data don’t exist anywhere else, this is what you have to do. While simple in concept (scrape the log, updated the database using ActiveRecord) this was very trick in implementation. I know my code could use some refactoring, but it is working pretty well right now.

So here is my code if anyone wants to do something similar.

I had a production log in the general format of:

Processing AttendanceController#update_exercises (for 76.105.103.226 at 2010-07-07 20:33:26) [POST]
  Session ID: 06af5da80fc41be243e42254cb776212
  Parameters: {"commit"=>"Record User Scores", "meeting_id"=>"3388", "exertions"=>{"exercise_note_437"=>"7:00 AM", "exercise_score_884"=>"6:52", "exercise_score_1120"=>"9:29", "exercise_score_1175"=>"13:21", "exercise_score_268"=>"8:07", "exercise_score_433"=>"6:29", "exercise_note_1200"=>"", "exercise_note_1190"=>"", "exercise_score_741"=>"11:02", "exercise_note_1191"=>"", "exercise_score_544"=>"8:37", "exercise_score_918"=>"***", "exercise_note_1124"=>"", "exercise_note_1069"=>"", "exercise_score_479"=>"9:18", "exercise_score_908"=>"8:04", "exercise_score_1012"=>"7:34", "exercise_score_105"=>"8:43", "exercise_note_417"=>"", "exercise_id"=>"25", "exercise_note_968"=>"", "exercise_note_913"=>"", "exercise_score_1200"=>"8:50", "exercise_note_1115"=>"", "exercise_score_437"=>"7:16", "exercise_score_1124"=>"9:32", "exercise_note_705"=>"", "exercise_score_1190"=>"12:26", "exercise_note_342"=>"", "exercise_score_1069"=>"9:34", "exercise_score_1191"=>"10:45", "exercise_note_1095"=>"", "exercise_note_872"=>"", "exercise_score_1115"=>"8:12", "exercise_score_417"=>"9:00", "exercise_note_884"=>"", "exercise_score_968"=>"9:02", "exercise_score_913"=>"7:59", "exercise_note_268"=>"", "exercise_note_433"=>"", "exercise_note_1130"=>"", "exercise_note_1031"=>"", "exercise_score_342"=>"9:26", "exercise_note_741"=>"", "exercise_note_1120"=>"", "exercise_note_544"=>"", "exercise_note_1175"=>"", "exercise_note_918"=>"", "exercise_score_1095"=>"6:59", "exercise_score_705"=>"7:29", "exercise_note_479"=>"", "exercise_note_908"=>"", "exercise_score_872"=>"8:25", "exercise_note_105"=>"", "exercise_score_1130"=>"8:14", "exercise_score_1031"=>"7:24", "exercise_note_1012"=>""}}
Redirected to actionindexid3388
Completed in 536ms (DB: 999) | 302 Found [http://www.fitwit.com/attendance/update_exercises]


Processing AttendanceController#index (for 76.105.103.226 at 2010-07-07 20:33:27) [GET]
  Session ID: 06af5da80fc41be243e42254cb776212
  Parameters: {"id"=>"3388"}
Rendering template within layouts/application
Rendering attendance/index
Completed in 195ms (View: 172, DB: 6) | 200 OK [http://www.fitwit.com/attendance/index/3388]

So I wrote this code to fix it. Basically, well, it’s complicated, if you have any questions, email me.

#!/usr/bin/env /Users/Tim/Sites/fitwit/script/runner

class CommandString
  attr_accessor :date, :method, :command

  def initialize(date, method, command)
    @date = date
    @method = method
    @command = command
  end
end

class ExerciseArray
  attr_accessor :exercise_id, :time_slot_id, :meeting_id, :values_hash

  def initialize(exercise_id, time_slot_id, meeting_id, values_hash)
    @exercise_id = exercise_id
    @time_slot_id = time_slot_id
    @meeting_id = meeting_id
    @values_hash = values_hash
  end
end

class RecoverData "Update", ("user_ids"=>[.*], "meeting_id"=>".*")/
        params = gen_hash("{#{$1}}")
        j+=1
        c = CommandString.new(@the_date,'take_attendance', params)
        commands <"(d+)", "commit"=>"Apply to ([ws]*)", "meeting_times"=>[(.*)], "id"=>"(d+)"}/
        boot_camp_id = $1.to_i
        if boot_camp_id = 40
          commit = "Apply to #{$2}"
          meeting_times = eval("[#{$3}]")
          time_slot_id = $4.to_i
          k+=1
          c = CommandString.new(@the_date,'manage_meetings', [time_slot_id, commit, meeting_times, boot_camp_id]) if boot_camp_id == 40
          commands <"Record User Scores", "meeting_id"=>"([0-9]+)", "exertions"=>({.*})}/
        meeting_id = $1.to_i
        #puts meeting_id
        attendees = gen_hash($2)
        i+=1
        c = CommandString.new(@the_date,'update_exercises', [meeting_id, attendees])
        commands < mt)
            ts.meetings < mt)
          @time_slot.meetings < user_id, :meeting_id => meeting_id) if submitted_user_new?(user_id, existing_attendee_ids)
          end
          # now deletes (for each existing user, see if they are excluded)
          existing_attendee_ids.each do |existing_id|
            if should_delete?(existing_id, post_user_ids)
              @meeting_user = MeetingUser.find_by_user_id_and_meeting_id(existing_id, meeting_id)
              @meeting_user.destroy
            end
          end
          puts "successful! attendance taken"
        else
          puts "bootcamp 41 or something, so no attendance taken"
        end
      else
        puts "attendance: #{meeting_id}"
      end
    else
      # need to raise exception
      puts "meeting id is nil for #{params.inspect}"
    end
  end

  def update_exercises(meeting_id, attendees)
    puts "updating exercises"
    # this is where we update our exercises which is the exertions table
    # the meeting _might_ not exist
    #begin
      if Meeting.exists?(meeting_id)
        meeting = Meeting.find(meeting_id)
        if meeting.time_slot.boot_camp.id == 40
          puts "!! found #{meeting_id}"
          users =  attendees.keys.map{|m| $1 if m.match(/exercise_score_(d+)$/) }.delete_if{|n| n.nil?} #  meeting.meeting_users
          exercise = Exercise.find(attendees["exercise_id"])
          users.each do |u_id|
            u = User.find(u_id)
            unless attendees["exercise_score_#{u.id}"].blank?
              ex = Exertion.new
              mus = MeetingUser.all(:conditions => ["meeting_id = ? and user_id = ?", meeting.id, u.id])
              if mus.empty? # then, we need to create an attendence
                mu = MeetingUser.create(:user_id => u.id, :meeting_id => meeting.id)
                puts "created attendance event for #{u.full_name}"
              else
                mu = mus.first
              end
              ex.meeting_user_id = mu.id
              ex.exercise_id = exercise.id
              ex.score = attendees["exercise_score_#{mu.user.id}"]
              raise "user: #{mu.user.id} mu: #{mu.id} ex: #{exercise.id}" if attendees["exercise_score_#{mu.user.id}"] == ''
              ex.notes = attendees["exercise_note_#{mu.user.id}"]
              ex.save
            end
          end
          puts "Recorded scores for #{exercise.name}"
          puts meeting.meeting_date
        else
          puts "wrong boot-camp #{meeting.time_slot.boot_camp.id}"
        end
      else
        puts "exercises:  #{meeting_id}"
      end
    #rescue

     # raise "bad: can't pull it off"
    #end
  end

  def submitted_user_new?(submitted_id,existing_ids)
    # if submitted_id is not in existing id's it is new
    !existing_ids.include?(submitted_id)
  end

  def should_delete?(existing_id,submitted_ids)
    #if existing id is not in submitted id it should be deleted
    !submitted_ids.include?(existing_id)
  end

end

BootCamp.find(40).time_slots.map{|ts| ts.meetings.destroy_all}

r = RecoverData.new()
commands = r.get_data
puts "about to run commands"
attendance_cursor = {98 => 0, 99 => 0, 100 => 0, 101 => 0}
exercise_cursor = {98 => 0, 99 => 0, 100 => 0, 101 => 0}
ea = []
commands.each do |c|
  case c.method
  when "take_attendance"
    user_id = c.command["user_ids"].first
    ts_all = User.find(user_id).registrations.map{|reg| reg.time_slot}
    ts = ts_all.select{|ts| ts if ts.boot_camp_id == 40}.first
    unless ts.nil?
      ts_id = ts.id
      bc = ts.boot_camp.id
      meeting_id = ts.meetings.map{|m| m.id}[[attendance_cursor[ts_id],23].min]
      #puts "meeting id is #{meeting_id}"
      attendance_cursor[ts_id]+=1
      puts "#{ts_id}, #{bc}, #{c.command["meeting_id"]}, #{c.date}, #{c.method}, #{c.command["user_ids"].to_sentence}"
      r.do_attendance(c.command, meeting_id)
    else
      puts "no timeslots for #{user_id}, but they are in bootcamps #{ts_all.map{|ts| ts.boot_camp.id}.to_sentence}"
    end
  when "manage_meetings"
    #  [time_slot_id, commit, meeting_times, boot_camp_id]
    if c.command.last == 40
      #puts "#{c.date}, #{c.method}, #{c.command.to_sentence}"
      r.manage_meetings(c.command[0], c.command[1], c.command[2], c.command[3])
    end
  when "update_exercises"
    # [meeting_id, attendees]
    params = c.command[1]
    params.first.first.match(/(d+)$/)
    user_id = $1 || 399 # this is to account for a wierd record that has an exercise if
    u = User.find(user_id)
    ts = u.registrations.map{|reg| reg.time_slot}.select{|ts| ts.boot_camp.id == 40}.first
    ex_id = params["exercise_id"]
    puts "!! #{ex_id}"
    unless ts.nil?
      ts_id = ts.id
      puts "time slot: #{ts_id}"
      m_index = [exercise_cursor[ts_id],23].min
#      puts "m index #{m_index}"
#      puts "meetings #{ts.meetings.inspect}"
      meeting_id = ts.meetings.map{|m| m.id}[m_index]
#      puts "meeting id: #{meeting_id}"
      exercise_cursor[ts_id]+=1
      # :exercise_id, :time_slot_id, :meeting_id, :values_hash
      ea << ExerciseArray.new(ex_id, ts_id, meeting_id, params)
      #puts "#{c.date}, #{c.method}, #{meeting_id}, #{c.command[1].inspect}"
      #r.update_exercises(meeting_id, c.command[1])
    else
      puts "all time slots are empty for bc 40 for user #{u.id} (they are in another boot_camp)"
    end
  end
end
puts "doing it"
ea.group_by{|e| [e.exercise_id, e.time_slot_id] }.each do |e|
  values_hash = {}
  meeting_id = e[1].first.meeting_id
  puts e.inspect
  e[1].each do |poopa|
    values_hash = values_hash.merge(poopa.values_hash)
  end
  puts "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
  puts "#{meeting_id}, #{values_hash.inspect}"
  puts "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
  r.update_exercises(meeting_id,values_hash)
end