# Assignment 3
All questions are weighted the same in this assignment. This assignment requires more individual learning then the last one did - you are encouraged to check out the [pandas documentation](http://pandas.pydata.org/pandas-docs/stable/) to find functions or methods you might not have used yet, or ask questions on [Stack Overflow](http://stackoverflow.com/) and tag them as pandas and python related. All questions are worth the same number of points except question 1 which is worth 17% of the assignment grade.
**Note**: Questions 3-13 rely on your question 1 answer.
import pandas as pd
import numpy as np
Load the energy data from the file `assets/Energy Indicators.xls`, which is a list of indicators of [energy supply and renewable electricity production](assets/Energy%20Indicators.xls) from the [United Nations](http://unstats.un.org/unsd/environment/excel_file_tables/2013/Energy%20Indicators.xls) for the year 2013, and should be put into a DataFrame with the variable name of **Energy**.
Keep in mind that this is an Excel file, and not a comma separated values file. Also, make sure to exclude the footer and header information from the datafile. The first two columns are unneccessary, so you should get rid of them, and you should change the column labels so that the columns are:
`['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable]`
Convert `Energy Supply` to gigajoules (**Note: there are 1,000,000 gigajoules in a petajoule**). For all countries which have missing data (e.g. data with "...") make sure this is reflected as `np.NaN` values.
Rename the following list of countries (for use in later questions):
```"Republic of Korea": "South Korea",
"United States of America": "United States",
"United Kingdom of Great Britain and Northern Ireland": "United Kingdom",
"China, Hong Kong Special Administrative Region": "Hong Kong"```
There are also several countries with numbers and/or parenthesis in their name. Be sure to remove these, e.g. `'Bolivia (Plurinational State of)'` should be `'Bolivia'`. `'Switzerland17'` should be `'Switzerland'`.
Next, load the GDP data from the file `assets/world_bank.csv`, which is a csv containing countries' GDP from 1960 to 2015 from [World Bank](http://data.worldbank.org/indicator/NY.GDP.MKTP.CD). Call this DataFrame **GDP**.
Make sure to skip the header, and rename the following list of countries:
```"Korea, Rep.": "South Korea",
"Iran, Islamic Rep.": "Iran",
"Hong Kong SAR, China": "Hong Kong"```
Finally, load the [Sciamgo Journal and Country Rank data for Energy Engineering and Power Technology](http://www.scimagojr.com/countryrank.php?cat
def answer_one():
energy = pd.read_excel('assets/Energy Indicators.xls')#导入
energy = energy[16:243]#删除标题行，保留数据行
energy = energy.drop(energy.columns[[0, 1]], axis=1)#删除前两列
energy.columns=['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']#更改列名
energy.replace('...', np.nan,inplace = True)#将…的数据改成nan
energy['Energy Supply'] *= 1000000#改单位
energy.replace(to_replace=' \(.*\)$',value='',regex=True,inplace=True)#去尾缀
energy.replace(to_replace='[\d]+$',value='',regex=True,inplace=True)#去尾缀
energy.replace(to_replace='^[\.]+$',value=np.nan,regex=True,inplace=True)#去尾缀
energy.replace({"Republic of Korea": "South Korea",#改名字
"United States of America": "United States",
"United Kingdom of Great Britain and Northern Ireland": "United Kingdom",
"China, Hong Kong Special Administrative Region": "Hong Kong"
},inplace = True)
GDP=pd.read_csv('assets/world_bank.csv',skiprows=4)#创建GDP的DataFrame
GDP.replace({"Korea, Rep.": "South Korea",
"Iran, Islamic Rep.": "Iran",
"Hong Kong SAR, China": "Hong Kong"
}, inplace=True)
GDP.rename(columns={'Country Name':'Country'},inplace=True)
ScimEn=pd.read_excel('assets/scimagojr-3.xlsx')#创建ScimEn的DataFrame
df = pd.merge(pd.merge(energy, GDP, on='Country'), ScimEn, on='Country')#根据country合并
df.set_index('Country',inplace=True)#改index
#改标题行
df = df[['Rank', 'Documents', 'Citable documents', 'Citations', 'Self-citations', 'Citations per document', 'H index', 'Energy Supply', 'Energy Supply per Capita', '% Renewable', '2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015']]
df = (df.loc[df['Rank'].isin([i for i in range(1, 16)])])#保留rank前15
df.sort_values("Rank", ascending=True, inplace=True)#排序
return df
answer_one()
assert type(answer_one()) == pd.DataFrame, "Q1: You should return a DataFrame!"
assert answer_one().shape == (15,20), "Q1: Your DataFrame should have 20 columns and 15 entries!"
### Question 2
The previous question joined three datasets then reduced this to just the top 15 entries. When you joined the datasets, but before you reduced this to the top 15 items, how many entries did you lose?
*This function should return a single number.*
%%HTML
<svg width="800" height="300">
<circle cx="150" cy="180" r="80" fill-opacity="0.2" stroke="black" stroke-width="2" fill="blue" />
<circle cx="200" cy="100" r="80" fill-opacity="0.2" stroke="black" stroke-width="2" fill="red" />
<circle cx="100" cy="100" r="80" fill-opacity="0.2" stroke="black" stroke-width="2" fill="green" />
<line x1="150" y1="125" x2="300" y2="150" stroke="black" stroke-width="2" fill="black" stroke-dasharray="5,3"/>
<text x="300" y="165" font-family="Verdana" font-size="35">Everything but this!</text>
</svg>
def answer_two():
# YOUR CODE HERE
energy = pd.read_excel('assets/Energy Indicators.xls')#导入
energy = energy[16:243]#删除标题行，保留数据行
energy = energy.drop(energy.columns[[0, 1]], axis=1)#删除前两列
energy.columns=['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']#更改列名
energy.replace('...', np.nan,inplace = True)#将…的数据改成nan
energy['Energy Supply'] *= 1000000#改单位
energy.replace(to_replace=' \(.*\)$',value='',regex=True,inplace=True)#去尾缀
energy.replace(to_replace='[\d]+$',value='',regex=True,inplace=True)#去尾缀
energy.replace(to_replace='^[\.]+$',value=np.nan,regex=True,inplace=True)#去尾缀
energy.replace({"Republic of Korea": "South Korea",#改名字
"United States of America": "United States",
"United Kingdom of Great Britain and Northern Ireland": "United Kingdom",
"China, Hong Kong Special Administrative Region": "Hong Kong"
},inplace = True)
GDP=pd.read_csv('assets/world_bank.csv',skiprows=4)#创建GDP的DataFrame
GDP.replace({"Korea, Rep.": "South Korea",
"Iran, Islamic Rep.": "Iran",
"Hong Kong SAR, China": "Hong Kong"
}, inplace=True)
GDP.rename(columns={'Country Name':'Country'},inplace=True)
ScimEn=pd.read_excel('assets/scimagojr-3.xlsx')#创建ScimEn的DataFrame
df = pd.merge(pd.merge(energy, GDP, on='Country'), ScimEn, on='Country')#根据country合并
df.set_index('Country',inplace=True)#改index
#改标题行
df = df[['Rank', 'Documents', 'Citable documents', 'Citations', 'Self-citations', 'Citations per document', 'H index', 'Energy Supply', 'Energy Supply per Capita', '% Renewable', '2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015']]
df = (df.loc[df['Rank'].isin([i for i in range(1, 16)])])#保留rank前15
union = pd.merge(pd.merge(energy, GDP, on='Country', how='outer'), ScimEn, on='Country', how='outer')
intersect = pd.merge(pd.merge(energy, GDP, on='Country'), ScimEn, on='Country')#默认是inner
return len(union)-len(intersect)
answer_two()
结果158
assert type(answer_two()) == int, "Q2: You should return an int number!"
### Question 3
What are the top 15 countries for average GDP over the last 10 years?
*This function should return a Series named `avgGDP` with 15 countries and their average GDP sorted in descending order.*
def answer_three():
# 计算10年的平均值，各年数据用apply()功能，利用匿名函数进行计算平均值。降序排列，可以用DataFrame中的sort_values()功能。
df=answer_one()
year=list(range(2006,2015))
year=[str(i) for i in year]
df['avgGDP']=df[year].apply(lambda x: np.nanmean(x),axis=1)
df=df.sort_values(ascending=False,by='avgGDP')
return df['avgGDP']
answer_three()
assert type(answer_three()) == pd.Series, "Q3: You should return a Series!"
### Question 4
By how much had the GDP changed over the 10 year span for the country with the 6th largest average GDP?
*This function should return a single number.*
def answer_four():#第六大GDP经济体10年变化值
Top15 = answer_one()
Top15["AvgGDP"] = answer_three()
Top15.sort_values("AvgGDP", ascending=False, inplace=True)
return abs(Top15.iloc[5]['2015'] - Top15.iloc[5]['2006'])
answer_four()
246702696075.3999
### Question 5
What is the mean energy supply per capita?
*This function should return a single number.*
### Question 5
What is the mean energy supply per capita?
*This function should return a single number.*
def answer_five():#人均能源供给的平均值
df=answer_one()
return df['Energy Supply per Capita'].mean()
answer_five()
157.6
### Question 6
What country has the maximum % Renewable and what is the percentage?
*This function should return a tuple with the name of the country and the percentage.*
def answer_six():
# 那个国家的’% Renewable’值最大，最大值是多少？这里我首先将DataFrame中的数据转化为float，然后用Series中的.idmax()找到了最大值所在国家
df=answer_one()
return (df['% Renewable'].astype(float).idxmax(),df['% Renewable'].astype(float).max())
answer_six()
('Brazil', 69.64803)
assert type(answer_six()) == tuple, "Q6: You should return a tuple!"
assert type(answer_six()[0]) == str, "Q6: The first element in your result should be the name of the country!"
### Question 7
Create a new column that is the ratio of Self-Citations to Total Citations.
What is the maximum value for this new column, and what country has the highest ratio?
*This function should return a tuple with the name of the country and the ratio.*
def answer_seven():
# 同6
df=answer_one()
df['ratio']=df['Self-citations']/df['Citations']
return (df['ratio'].idxmax(),df['ratio'].max())
answer_seven()
('China', 0.6893126179389422)
### Question 8
Create a column that estimates the population using Energy Supply and Energy Supply per capita.
What is the third most populous country according to this estimate?
*This function should return the name of the country*
def answer_eight():
# Energy Supply’/‘Energy Supply per capita’
df=answer_one()
df['population']=df['Energy Supply']/df['Energy Supply per Capita']
df.sort_values(by='population',ascending=False,inplace=True)
return df.iloc[2].name
answer_eight()
'United States'
### Question 9
Create a column that estimates the number of citable documents per person.
What is the correlation between the number of citable documents per capita and the energy supply per capita? Use the `.corr()` method, (Pearson's correlation).
*This function should return a single number.*
*(Optional: Use the built-in function `plot9()` to visualize the relationship between Energy Supply per Capita vs. Citable docs per Capita)*
def answer_nine():
# YOUR CODE HERE
df=answer_one()
df['population']=df['Energy Supply']/df['Energy Supply per Capita']
df['citable documents per capita']=df['Citable documents']/df['population']
return df['citable documents per capita'].astype(float).corr(df['Energy Supply per Capita'].astype(float))
answer_nine()
def plot9():
import matplotlib as plt
%matplotlib inline
Top15 = answer_one()
Top15['PopEst'] = Top15['Energy Supply'] / Top15['Energy Supply per Capita']
Top15['Citable docs per Capita'] = Top15['Citable documents'] / Top15['PopEst']
Top15.plot(x='Citable docs per Capita', y='Energy Supply per Capita', kind='scatter', xlim=[0, 0.0006])
assert answer_nine() >= -1. and answer_nine() <= 1., "Q9: A valid correlation should between -1 to 1!"
### Question 10
Create a new column with a 1 if the country's % Renewable value is at or above the median for all countries in the top 15, and a 0 if the country's % Renewable value is below the median.
*This function should return a series named `HighRenew` whose index is the country name sorted in ascending order of rank.*
def answer_ten():
# 求得中位数，大于等于中位数则返回1，否则返回0
df=answer_one()
df['flagRenew']= df['% Renewable']-df['% Renewable'].median()
df['HighRenew']= df['flagRenew'].apply(lambda x:1 if x>=0 else 0)
return df['HighRenew']
raise NotImplementedError()
assert type(answer_ten()) == pd.Series, "Q10: You should return a Series!"
### Question 11
Use the following dictionary to group the Countries by Continent, then create a DataFrame that displays the sample size (the number of countries in each continent bin), and the sum, mean, and std deviation for the estimated population of each country.
```python
ContinentDict = {'China':'Asia',
'United States':'North America',
'Japan':'Asia',
'United Kingdom':'Europe',
'Russian Federation':'Europe',
'Canada':'North America',
'Germany':'Europe',
'India':'Asia',
'France':'Europe',
'South Korea':'Asia',
'Italy':'Europe',
'Spain':'Europe',
'Iran':'Asia',
'Australia':'Australia',
'Brazil':'South America'}
```
*This function should return a DataFrame with index named Continent `['Asia', 'Australia', 'Europe', 'North America', 'South America']` and columns `['size', 'sum', 'mean', 'std']`*
def region(row):
ContinentDict = {'China':'Asia',
'United States':'North America',
'Japan':'Asia',
'United Kingdom':'Europe',
'Russian Federation':'Europe',
'Canada':'North America',
'Germany':'Europe',
'India':'Asia',
'France':'Europe',
'South Korea':'Asia',
'Italy':'Europe',
'Spain':'Europe',
'Iran':'Asia',
'Australia':'Australia',
'Brazil':'South America'}
#print(row.name)
for key,values in ContinentDict.items():
if row.name == key:
row['region']=values
return row
def answer_eleven():
df=answer_one()
df['population']=df['Energy Supply']/df['Energy Supply per Capita']
df=df.apply(region,axis=1)
new_df=df.groupby('region',axis=0).agg({'population':(np.size,np.nansum,np.nanmean,np.nanstd)})
#new_df['size']=df.groupby('region',axis=0).size()
new_df.columns=['size','sum','mean','std']
#print(new_df)
return new_df
answer_eleven()
assert type(answer_eleven()) == pd.DataFrame, "Q11: You should return a DataFrame!"
assert answer_eleven().shape[0] == 5, "Q11: Wrong row numbers!"
assert answer_eleven().shape[1] == 4, "Q11: Wrong column numbers!"
### Question 12
Cut % Renewable into 5 bins. Group Top15 by the Continent, as well as these new % Renewable bins. How many countries are in each of these groups?
*This function should return a Series with a MultiIndex of `Continent`, then the bins for `% Renewable`. Do not include groups with no countries.*
def region(row):
ContinentDict = {'China':'Asia',
'United States':'North America',
'Japan':'Asia',
'United Kingdom':'Europe',
'Russian Federation':'Europe',
'Canada':'North America',
'Germany':'Europe',
'India':'Asia',
'France':'Europe',
'South Korea':'Asia',
'Italy':'Europe',
'Spain':'Europe',
'Iran':'Asia',
'Australia':'Australia',
'Brazil':'South America'}
#print(row.name)
for key,values in ContinentDict.items():
if row.name == key:
row['Continent']=values
return row
def answer_twelve():
df=answer_one()
df=df.apply(region,axis=1)
df['% Renewable']=pd.cut(df['% Renewable'],5)
new_df=df.groupby(['Continent','% Renewable'])['Continent'].agg(np.size).dropna()
#new_df=new_df.set_index(['Continent','intervals'])
return new_df
answer_twelve()
### Question 13
Convert the Population Estimate series to a string with thousands separator (using commas). Use all significant digits (do not round the results).
e.g. 12345678.90 -> 12,345,678.90
*This function should return a series `PopEst` whose index is the country name and whose values are the population estimate string*
def answer_thirteen():
df=answer_one()
df['population']=df['Energy Supply']/df['Energy Supply per Capita']
return df['population'].apply('{:,}'.format)
answer_thirteen()
