Rick and Morty by the numbers

Rick and Morty is quickly becoming a cult classic. With the last episode coming out on October 4, 2015, fans have been starving for new episodes. On April fools, the creators of the show surprised fans with a new episode from the long awaited third season.

After seeing the new episode, I decided to do an analysis of Rick and Morty. This analysis is heavily influenced by Todd Schneider's blog post analyzing The Simpsons (I highly recommend you check it out).

Rick and Morty Ratings

First, I wrote code to gather data from IMDb. The average rating of each episode is above 8; except season 2 episode 8: "Interdimensional Cable 2: Tempting Fate". In this episode, Jerry is faced with keeping his penis or have it surgically removed to save the life of an alien leader. The other main characters sit in the lobby of the hospital and watch interdimensional cable. The shows on cable are short clips and are extremely random. Fans accustomed to the great banter between Rick and Morty get almost none this episode.

The family dynamic

Then I gathered and parsed scripts from RickAndMorty.wikia.com. Unfortunately, only about half the scripts are online. I took the average number of words each member of the family spoke per episode. I left out Mr. Poopybutthole and Sleepy Gary because they are more "like family" than family. 

Rick is the center of the show and has by far the most lines. Next is Jerry, this is because in the episode "The Wedding Squanchers", Morty has almost no lines. Since I only have a portion of the scripts, this lowers Morty's average number of lines to below Jerry. When more transcripts are uploaded to the Wikia, I will rerun my analysis and update the post. 

Recurring Characters:

Justin Roiland and Dan Harmon have a special ability to write unique characters that audiences love. Below is a chart of recurring characters from the show. This data is from the credits. Sometimes characters are not in the episode, but are still in the credits. When this happens, the character is represented with a gray box.

After waiting 2 years, I can't wait to watch more of season 3 and I hope Justin and Dan choose to include more of Snuffles. I think it could be developed into a very satisfying plot for people of all ages.

---

Note:

• The graphs were made using ggplot2 and the data was gathered using rvest. (God bless Hadley Wickham)

College Majors with Fewer Women tend to have Larger Pay Gaps

The Gender Pay Gap has become a key talking point for many in politics in the past few years. It is commonly quoted that women make only 77¢ for every $1 of their male counterparts. This statistic is calculated by taking the average salary of women and comparing it to the average salary of men. This does not control for job title, degree, regional difference, company size, or hours worked. After controlling for these factors, the gender pay gap drops from 77% to 98% (Payscale.com does a fantastic analysis and break down of the gender pay gap. I highly recommend you check it out.)

Using the salaries that controlled for these factors from the Payscale report, I decided to do further analysis on the pay gap. I wanted to know how the percentage of women in a field effected the pay difference between men and women. Since many companies try to hire equal amounts of men and women for similar roles and there are significantly fewer women in science and engineering, I assumed these few women would be able to demand a higher salary because of the scarcity. However, I found the opposite.

The interactive graphic below shows that college majors with fewer women tend to have higher differences in pay. (The graphics may be difficult to see on mobile, please switch to a desktop or use the mobile-friendly version)

Two of the largest outliers are Nursing and Accounting. Nursing has 92.3% women and men still make about $2400 more on average. Accounting has 52.1% women and a pay difference of +$2400 for men. One point of interest is elementary education where women make 1.4% more than men.

The following graph shows the same results as above, but instead of absolute difference in dollars, the commonly used metric of women's salaries as a percentage of men's salaries is plotted.



Any guess as to why this occurs would largely be speculative, but I imagine a “boy’s club” mentality could be to blame where men like to hire men and are willing to even pay more to do so.

The links to the data sources can be found below and as always, add any suggestions in the comments.

---

Data Source:

[1] http://www.payscale.com/career-news/2009/12/do-men-or-women-choose-majors-to-maximize-income
[2] https://docs.google.com/spreadsheets/d/1FDrXUk4t-RQekuKotqMD7pyGFmylHip-xarOawcVvqk/edit#gid=0

A Look at Lynching in the United States

With the current racial tension within the United States and the Halloween season, I have seen many posts on Facebook showing black mannequins and other human-like figures hanging from trees. These images spark debate on whether these are directly racist acts or just distasteful mistakes. The comment sections are filled with pseudo-lynching-experts defending both sides.

So I decided to do an analysis on lynching within the United States. I found data on lynching from the Tuskegee Institute [1-2]. This data set contains information on lynchings from 1882-1962. I tried to find lynching statistics from before 1882 and found, during slavery, lynching Africans was fairly uncommon due to slave owners having a vested interest in keeping the slave alive.

The first bit of information I found shocking was whites made up 27% of lynching victims. However, the percentage of whites vs. blacks changes wildly from state to state. To understand this the percentage of black victims were plotted by state. As can be assumed, the deep south had the highest percentage of blacks followed by the rest of the south.

However, a look at the total number of lynchings showed states in the deep south did not offend evenly. Mississippi, Georgia, and Texas had the most lynchings (581, 531, and 493 respectfully) and then there was a large drop off to Louisiana and Alabama at 391 and 347. New Hampshire had zero lynchings and Delaware, Maine, and Vermont had one. New York and New Jersey only had two.

Over time, the number of people lynched has dropped to near zero. The next graph shows how lynching has decreased over time for whites and blacks. Black lynchings have about a 30 year lag behind the decrease of white lynchings.

In the first few years of the above chart, whites out number blacks in the number of lynchings. Below is a graph showing the percentage breakdown by year. Whites out number blacks for 4 years and then blacks out number whites for 60 years. In the last few years in the data set, whites exceed blacks. When taking into account how infrequent lynchings where in these last few years, two white people being lynched can account for 66.6% of lynchings.

I hope you learned something new, reading this analysis and as always, please feel free to add ideas for additional graphs or analysis in the comment section.

---

Data Link and Notes:

[1] - http://law2.umkc.edu/faculty/projects/ftrials/shipp/lynchingsstate.html
[2] - http://law2.umkc.edu/Faculty/projects/ftrials/shipp/lynchingyear.html

Salaries of Presidential Primary Voters by Candidate and State

The data used in this post comes from FiveThirtyEight and are put into easy to understand graphics. The first graphic shows the average salary of supports of each candidate by state. The states are ranked by highest average salary, Maryland, to the state with the lowest salary, Mississippi. The red line shows the average salary for each state. The first obvious conclusion is that John Kasich supporters make about $5-10 thousand more than supporters of other candidates. Second, in low salary states Clinton and Sanders supporter's have similar salaries, but when looking at higher salary states, Clinton supporters' salaries are even with Trump and Cruz supporters' salaries.

The following plot shows the distribution of average salaries for each Presidential Candidate. Again we see similar trends as before with Kasich having a high average income and Clinton having a mix of low and high income supporters.

The last plot shows the relationship between the average salary of a state and the average salary of a candidate's supporters. The red line is a perfect 1 to 1 ratio and the closer a candidate is to the red line the closer the candidate's supporters are to having the same salary as the average person of that state. The reason for almost all the dots falling above the line is because people with below average salaries are less likely to vote.

If you have any suggestions for plots using this data, please share in the comment section.

---

Note:

[1] All states are not included because all states have not held elections yet.

Introduction to R: For Beginners Who Want to be Intermediate

This semester, I am managing an undergraduate on our research team. She has a basic engineering statistics background with a little experience doing analysis in Excel and understands some basic programming concepts. For her project with our group, she will have to learn R. 

I tried to compile a list of high quality videos and tutorials. While searching, I was surprised by two things: the increase in quantity of resources since when I started to learn R and the giant gap between beginner tutorials and the skills needed to be proficient in R. Many beginner tutorials explained how to assign or print out a variable, but no introduction to real analysis or real-world application of these skills. Also many of the tutorials only included basic R functions, for example plot() instead of ggplot2(), which any good Hadley Wickham worshipper knows is akin to sin (For those of you are just starting out, Hadley Wickham created ggplot2 and some of the most used R packages of all time). So I decided to post my compilation of tutorials and videos that will take you from a person that knows nothing about R to an intermediate R programmer in a smooth transition.

Step 1: Download and Setup

Download R  -   LINK

This is the basic program to run R on your computer.

Download RStudio  -   LINK

This is an IDE or Integrated development environment for R. Always open your R files into RStudio and not basic R because RStudio helps track variables, stores past graphs and has many other features that you will use heavily in analyzing data.

Step 2: Understand the Basic

Follow this tutorial for R  -  LINK

This is all online but it gets the basics of the language down.

Do the Introduction to R tutorial and Data Manipulation in R with dplyr. If you have trouble finding them in the site, the links to each class are here.

https://www.datacamp.com/courses/free-introduction-to-r

https://www.datacamp.com/courses/dplyr-data-manipulation-r-tutorial

Step 2.5: Really a tip more than a step

The rest of the steps will be inside RStudio. To run a line of code, click in the line of code you wish to run and press Crt +Enter for Windows or Command + Enter for Mac. The line should drop down to the console window and run. This will make more sense once you watch the later videos.

Step 3: Introduction to R and RStudio

https://www.youtube.com/watch?v=lVKMsaWju8w

Step 4: Data Visualization and Generating Graphs

R is made of groups of functions called “packages”. There are packages for everything in R from accessing databases to creating websites to creating predictive models to generating plots.

The most common package for data analysis is ggplot2. GGplot2 allows R users to quickly plot data stored in data frames. The following video steps you through using qplot (which stands for quick plot and is ggplot2 most popular function).

Introduction to R Programming - Module 8 (qplot)  -  LINK

The data for this video can be downloaded here and is called “bank.zip”: http://mlr.cs.umass.edu/ml/machine-learning-databases/00222/

The file you want for this video is bank-full.csv.

In the video, replace the line:

attach(bank.marketing)

with the following code:

install.packages("ggplot2")

library(ggplot2)

file_path <- file.choose()

bank.marketing <- read.csv2(file_path)

This is how you read a csv data file separated by “;” into R, if it was separated by commas then the command would be read.csv(). Now the data in the csv selected is in the data frame “bank.marketing”.

The command file.choose() is used when the path to the file is unknown and will pop up a file search for you to select the desired file.

Step 5: Intro to Data Science

This is a great video for learning data analysis in R. There are three parts, I would recommend looking at all three parts.

https://www.youtube.com/watch?v=32o0DnuRjfg
https://www.youtube.com/watch?v=u6sahb7Hmog

https://www.youtube.com/watch?v=aMV_6LmCs4Q

Step 6: Intro to Web scraping

R Web Page Scraping Example Video

https://www.youtube.com/watch?v=GxQLPNtJiMM

Web scraping is super easy in R and can allow you to grab data others can not. It is often missed in intro R tutorials which is a shame. 

Step 7: Introduction to Machine Learning with Random Forest

A quick introduction tutorial to machine learning algorithms and best practices.

http://www.hallwaymathlete.com/2016/05/introduction-to-machine-learning-with.html

Step 8: Machine Learning Course

Hands down the best machine learning course on the planet. 

http://www.r-bloggers.com/in-depth-introduction-to-machine-learning-in-15-hours-of-expert-videos/

At this point, you are a full data scientist and you can gather and analyze any data on the web!