Pandas
Libraries
16 Underrated Pandas Series Methods And When To Use Them
In this article, we’re going to explore some lesser-known but very useful pandas methods for manipulating Series objects. Some of these methods are related only to Series, the others — both to Series and DataFrames, having, however, specific features when used with both structure types.1. is_uniqueAs its name sugests, this method checks if all the values of a Series are unique:import pandas as pd
print(pd.Series([1, 2, 3, 4]).is_unique)
print(pd.Series([1, 2, 3, 1]).is_unique)
Output:
True
False
2 & 3. is_monotonic and is_monotonic_decreasingWith these 2 methods, we can check if the values of a Series are in ascending/descending order:print(pd.Series([1, 2, 3, 8]).is_monotonic)
print(pd.Series([1, 2, 3, 1]).is_monotonic)
print(pd.Series([9, 8, 4, 0]).is_monotonic_decreasing)
Output:
True
False
TrueBoth methods work also for a Series with string values. In this case, Python uses a lexicographical ordering under the hood, comparing two subsequent strings character by character. It’s not the same as just an alphabetical ordering, and actually, the example with the numeric data above is a particular case of such an ordering. As the Python documentation says,Lexicographical ordering for strings uses the Unicode code point number to order individual characters.In practice, it mainly means that the letter case and special symbols are also taken into account:print(pd.Series(['fox', 'koala', 'panda']).is_monotonic)
print(pd.Series(['FOX', 'Fox', 'fox']).is_monotonic)
print(pd.Series(['*', '&', '_']).is_monotonic)
Output:
True
True
FalseA curious exception happens when all the values of a Series are the same. In this case, both methods return True:print(pd.Series([1, 1, 1, 1, 1]).is_monotonic)
print(pd.Series(['fish', 'fish']).is_monotonic_decreasing)
Output:
True
TrueAlso Read: 4 Must-Know Python Pandas Functions for Time Series Analysis4. hasnansThis method checks if a Series contains NaN values:import numpy as np
print(pd.Series([1, 2, 3, np.nan]).hasnans)
print(pd.Series([1, 2, 3, 10, 20]).hasnans)
Output:
True
False5. emptySometimes, we might want to know if a Series is completely empty, not containing even NaN values:print(pd.Series().empty)
print(pd.Series(np.nan).empty)
Output:
True
FalseA Series can become empty after some manipulations with it, for example, filtering:s = pd.Series([1, 2, 3])
s[s > 3].empty
Output:
True
6 & 7. first_valid_index() and last_valid_index()These 2 methods return index for first/last non-NaN value and are particularly useful for Series objects with many NaNs:print(pd.Series([np.nan, np.nan, 1, 2, 3, np.nan]).first_valid_index())
print(pd.Series([np.nan, np.nan, 1, 2, 3, np.nan]).last_valid_index())
Output:
2
4If all the values of a Series are NaN, both methods return None:print(pd.Series([np.nan, np.nan, np.nan]).first_valid_index())
print(pd.Series([np.nan, np.nan, np.nan]).last_valid_index())
Output:
None
None
8. truncate()This method allows truncating a Series before and after some index value. Let’s truncate the Series from the previous section leaving only non-NaN values:s = pd.Series([np.nan, np.nan, 1, 2, 3, np.nan])
s.truncate(before=2, after=4)
Output:
2 1.0
3 2.0
4 3.0
dtype: float64The original index of the Series was preserved. We may want to reset it and also to assign the truncated Series to a variable:s_truncated = s.truncate(before=2, after=4).reset_index(drop=True)
print(s_truncated)
Output:
0 1.0
1 2.0
2 3.0
dtype: float64Also Read: Pandas vs SQL. When Data Scientists Should Use One Over the Other9. convert_dtypes()As the pandas documentation says, this method is used toConvert columns to best possible dtypes using dtypes supporting pd.NA.If to consider only Series objects and not DataFrames, the only application of this method is to convert all nullable integers (i.e. float numbers with a decimal part equal to 0, such as 1.0, 2.0, etc.) back to “normal” integers. Such float numbers appear when the original Series contains both integers and NaN values. Since NaN is a float in numpy and pandas, it leads to the whole Series with any missing values to become of float type as well.Let’s take a look at the example from the previous section to see how it works:print(pd.Series([np.nan, np.nan, 1, 2, 3, np.nan]))
print('\n')
print(pd.Series([np.nan, np.nan, 1, 2, 3, np.nan]).convert_dtypes())
Output:
0 NaN
1 NaN
2 1.0
3 2.0
4 3.0
5 NaN
dtype: float64
0 <NA>
1 <NA>
2 1
3 2
4 3
5 <NA>
dtype: Int64
10. clip()We can clip all the values of a Series at input thresholds (lower and upper parameters):s = pd.Series(range(1, 11))
print(s)
s_clipped = s.clip(lower=2, upper=7)
print(s_clipped)
Output:
0 1
1 2
2 3
3 4
4 5
5 6
6 7
7 8
8 9
9 10
dtype: int64
0 2
1 2
2 3
3 4
4 5
5 6
6 7
7 7
8 7
9 7
dtype: int64
11. rename_axis()In the case of a Series object, this method sets the name of the index:s = pd.Series({'flour': '300 g', 'butter': '150 g', 'sugar': '100 g'})
print(s)
s=s.rename_axis('ingredients')
print(s)
Output:
flour 300 g
butter 150 g
sugar 100 g
dtype: object
ingredients
flour 300 g
butter 150 g
sugar 100 g
dtype: object
12 & 13. nsmallest() and nlargest()These 2 methods return the smallest/largest elements of a Series. By default, they return 5 values, in ascending order for nsmallest() and in descending - for nlargest().s = pd.Series([3, 2, 1, 100, 200, 300, 4, 5, 6])
s.nsmallest()
Output:
2 1
1 2
0 3
6 4
7 5
dtype: int64It’s possible to specify another number of the smallest/largest values to be returned. Also, we may want to reset the index and assign the result to a variable:largest_3 = s.nlargest(3).reset_index(drop=True)
print(largest_3)
Output:
0 300
1 200
2 100
dtype: int64Also Read: Pandas vs SQL. When Data Scientists Should Use One Over the Other14. pct_change()For a Series object, we can calculate percentage change (or, more precisely, fraction change) between the current and a prior element. This approach can be helpful, for example, when working with time series, or for creating a waterfall chart in % or fractions.s = pd.Series([20, 33, 14, 97, 19])
s.pct_change()
Output:
0 NaN
1 0.650000
2 -0.575758
3 5.928571
4 -0.804124
dtype: float64To make the resulting Series more readable, let’s round it:s.pct_change().round(2)
Output:
0 NaN
1 0.65
2 -0.58
3 5.93
4 -0.80
dtype: float64
15. explode()This method transforms each list-like element of a Series (lists, tuples, sets, Series, ndarrays) to a row. Empty list-likes will be transformed in a row with NaN. To avoid repeated indices in the resulting Series, it’s better to reset index:s = pd.Series([[np.nan], {1, 2}, 3, (4, 5)])
print(s)
s_exploded = s.explode().reset_index(drop=True)
print(s_exploded)
Output:
0 [nan]
1 {1, 2}
2 3
3 (4, 5)
dtype: object
0 NaN
1 1
2 2
3 3
4 4
5 5
dtype: object
16. repeat()This method is used for consecutive repeating each element of a Series a defined number of times. Also in this case, it makes sense to reset index:s = pd.Series([1, 2, 3])
print(s)
s_repeated = s.repeat(2).reset_index(drop=True)
print(s_repeated)
Output:
0 1
1 2
2 3
dtype: int64
0 1
1 1
2 2
3 2
4 3
5 3
dtype: int64If the number of repetitions is assigned to 0, an empty Series will be returned:s.repeat(0)
Output:
Series([], dtype: int64)
ConclusionTo sum up, we investigated 16 rarely used pandas methods for working with Series and some of their application cases. If you know some other interesting ways to manipulate pandas Series, you’re very welcome to share them in the comments.Thanks for reading!Also read: Using Python And Pandas Datareader to Analyze Financial Data
Libraries
Pandas
4 Must-Know Python Pandas Functions for Time Series Analysis
Time series data consists of data points attached to sequential time stamps. Daily sales, hourly temperature values, and second-level measurements in a chemical process are some examples of time series data.Time series data has different characteristics than ordinary tabular data. Thus, time series analysis has its own dynamics and can be considered as a separate field. There are books over 500 pages to cover time series analysis concepts and techniques in depth.Pandas was created by Wes Mckinney to provide an efficient and flexible tool to work with financial data which is kind of a time series. In this article, we will go over 4 Pandas functions that can be used for time series analysis.We need data for the examples. Let’s start with creating our own time series data.import numpy as np
import pandas as pd
df = pd.DataFrame({
"date": pd.date_range(start="2020-05-01", periods=100, freq="D"),
"temperature": np.random.randint(18, 30, size=100) +
np.random.random(100).round(1)
})
df.head()(image by author)We have created a data frame that contains temperature measurements during a period of 100 days. The date_range function of Pandas can be used for generating a date range with customized frequency. The temperature values are generated randomly using Numpy functions.We can now start on the functions.1. ShiftIt is a common operation to shift time series data. We may need to make a comparison between lagged or lead features. In our data frame, we can create a new feature that contains the temperature of the previous day.df["temperature_lag_1"] = df["temperature"].shift(1)
df.head()(image by author)The scalar value passed to the shift function indicates the number of periods to shift. The first row of the new column is filled with NaN because there is no previous value for the first row.The fill_value parameter can be used for filling the missing values with a scalar. Let’s replace the NaN with the average value of the temperature column.df["temperature_lag_1"] = df["temperature"]\
.shift(1, fill_value = df.temperature.mean())
df.head()(image by author)If you are interested in the future values, you can shift backwards by passing negative values to the shift function. For instance, “-1” brings the temperature in the next day.2. ResampleAnother common operation performed on time series data is resampling. It involves in changing the frequency of the periods. For instance, we may be interested in the weekly temperature data rather than daily measurements.The resample function creates groups (or bins) of a specified internal. Then, we can apply aggregation functions to the groups to calculate the value based on resampled frequency.Let’s calculate the average weekly temperatures. The first step is to resample the data to week level. Then, we will apply the mean function to calculate the average.df_weekly = df.resample("W", on="date").mean()
df_weekly.head()(image by author)The first parameter specifies the frequency for resampling. “W” stands for week, surprisingly. If the data frame does not have a datetime index, the column that contains the date or time related information needs to be passed to the on parameter.3. AsfreqThe asfreq function provides a different technique for resampling. It returns the value at the end of the specified interval. For instance, asfreq(“W”)returns the value on the last day of each week.In order to use the asfreq function, we should set the date column as the index of the data frame.df.set_index("date").asfreq("W").head()(image by author)Since we are getting a value at a specific day, it is not necessary to apply an aggregation function.4. RollingThe rolling function can be used for calculating moving average which is a highly common operation for time series data. It creates a window of a particular size. Then, we can use this window to make calculations as it rolls through the data points.The figure below explains the concept of rolling.(image by author)Let’s create a rolling window of 3 and use it to calculate the moving average.df.set_index("date").rolling(3).mean().head()(image by author)For any day, the values show the average of the day and the previous 2 days. The values of the first 3 days are 18.9, 23.8, and 19.9. Thus, the moving average on the third day is the average of these values which is 20.7.The first 2 values are NaN because they do not have previous 2 values. We can also use this rolling window to cover the previous and next day for any given day. It can be done by setting the center parameter as true.df.set_index("date").rolling(3, center=True).mean().head()(image by author)The values of the first 3 days are 18.9, 23.8, and 19.9. Thus, the moving average in the second day is the average of these 3 values. In this setting, only the first value is NaN because we only need 1 previous value.ConclusionWe have covered 4 Pandas functions that are commonly used in time series analysis. Predictive analytics is an essential part of data science. Time series analysis is at the core of many problems that predictive analytics aims to solve. Hence, if you plan to work on predictive analytics, you should definitely learn how to handle time series data.Thank you for reading. Please let me know if you have any feedback.Soner Yıldırım
Libraries
Deployment
Data Scientists Without Data Engineering Skills Will Face the Harsh Truth
OPINION.You have probably read an article about the difference between a data scientist and a data engineer. I always thought the distinction was clear. Data engineers make the data ready for use and then data scientists work on that data.However, my opinion on this distinction has changed dramatically after I started working as a data scientist.Photo by Ben White on UnsplashYou have probably read an article about the difference between a data scientist and a data engineer. I always thought the distinction was clear. Data engineers make the data ready for use and then data scientists work on that data.Everything in data science starts with data. Your machine learning model is just as good as the data fed into it. Garbage in, garbage out! A data scientist cannot do some magic to create a valuable product without proper data.The proper data is not always readily available for data scientists. In most cases, it will the responsibility of the data scientist to convert the raw data to a proper format.Unless you work for a big tech company that has separate teams of data engineers and data scientists, you should possess the ability and skills to handle some data engineering tasks. These tasks cover a broad range of operations and I will elaborate on this in the remaining part of the article.What is the difference anyway?I would like to state my opinion on the relationship between the job of a data engineer and a data scientist.A data engineer is a data engineer. A data scientist should be both a data scientist and a data engineer.It may seem like an arguable statement. However, I would like to emphasize that my opinion was different before I started working as a data scientist. I used to think of data engineers and data scientists as separate entities.In the remaining part of the article, I will try to explain what I mean by a data scientist should be both a data scientist and a data engineer.For instance, data engineers do a set of operations known as ETL (extract, transform, load). It covers the procedures for collecting data from one or more sources, apply some transformations, and then load into a different source.I would definitely not be surprised if a data scientist is expected to perform ETL operations. Data science is still evolving and most companies do not have clearly separated data engineer and data scientist roles. As a result, a data scientist should be able to perform some data engineering tasks.If you expect to only work on running machine learning algorithms with ready-to-use data, you will face the harsh truth soon after you start working as a data scientist.You may have to write some stored procedures in SQL to preprocess the client data. It is also possible that you receive the client data from a few different sources. It will be your job to extract and combine them. Then, you will need to load them into a single source. In order to write efficient stored procedures, you need extensive SQL skills.The transform part of ETL procedures involves in many data cleaning and manipulation steps. SQL may not be the best choice if you work with large-scale data. Distributed computing is a better alternative in such cases. Therefore, a data scientist should also be familiar with distributed computing.Your best friend in distributed computing might be Spark. It is an analytics engine used for large-scale data processing. We can distribute both data and computations over clusters to achieve a substantial performance increase.If you are familiar with Python and SQL, you won’t have hard time getting used to Spark. You can use Spark features with PySpark which is a Python API for Spark.Read also: A Beginner’s Guide to Apache SparkWhen it comes to work with clusters, the optimal environment is the cloud. There are various cloud providers but AWS, Azure, and Google Cloud Platform (GCP) lead the way.Although the PySpark code is the same for all cloud providers, how you setup the environment and create clusters change between them. They allow for creating clusters using both scripts or the user interface.Distributed computing over clusters is a whole different world. It is nothing like doing analysis in your computer. It has very different dynamics. Evaluating cluster performance and choosing the optimal number of workers for a cluster will be your predominant concerns.Read also:* The Full Stack Data Scientist* Everything a Data Scientist Should Know About Data Management
Libraries
Customize your Jupyter Notebooks
IntroductionJupyter Notebook is a great programming environment and often the most popular choice for data scientists or data analysts that are coding in python. Unfortunately, its default settings do not allow the level of customization that you have with standard programming environments such as PyCharm or similar tools.Jupyter Notebooks themes are trying to diminish this gap and allow you to make the notebook a bit prettier and also more functional using the themes. In this article, I will walk you through the installation process of Jupyter Notebook themes and show you some of their most important features.InstallationJupyter Notebook themes is an open-source library and can be installed with pip install. Just type the following code in a command line:pip install jupyterthemesThis should trigger the installation of the latest version. Once done you should be able to switch between themes, adjust fonts used in the notebooks, or customize the style of the plots. We will go through these features in detail in the next sections.Changing themesAfter installation, you can launch Jupyter Notebooks as normal and inspect the themes from within the notebook itself.In order to list all possible themes, you can use the following code:!jt -lAs you can see there are currently nine themes available. In order to switch themes you can use this command:!jt -t <theme_name>Let’s choose onedork theme.!jt -t onedorkYou will see that the theme does not change immediately. Some people report that they can reload the page and see the effect after that. From my personal experience, I have to restart Jupyter Notebook in order for the theme to change. Just stop the notebook and launch it again. This is how it should look like with onedork theme when it is loaded.Now, you can play with different themes and choose your favorite one.What you will notice is that some parts of the standard GUI are not visible by default in the theme settings. I am referring to the part pictured below.In order to switch the theme but keeping the standard GUI look, you could use the following variation of the code.!jt -t <theme_name> -T -N -klWith onedork theme, these would like that.!jt -t onedork -T -N -klRestarting Jupyter Notebooks should give you the result similar to the screenshot below.In order to restore the notebook to its default settings, you can use this code.!jt -rNote that I have shown the commands being executed in the Jupyter notebook itself but you can use them without the exclamation mark in the terminal window as well.Setting up the graphing styleOnce you are using the themes you will notice that the graphs created with Matplotlib library do not look the best. For example, this is a simple code to create a line chart.import matplotlib.pyplot as plt
%matplotlib inline
bp_x = np.linspace(0, 2*np.pi, num=40, endpoint=True)
bp_y = np.sin(bp_x)
# Make the plot
plt.plot(bp_x, bp_y, linewidth=3, linestyle="--",
color="blue", label=r"Legend label $\sin(x)$")
plt.xlabel(r"Description of $x$ coordinate (units)")
plt.ylabel(r"Description of $y$ coordinate (units)")
plt.title(r"Title here (remove for papers)")
plt.xlim(0, 2*np.pi)
plt.ylim(-1.1, 1.1)
plt.legend(loc="lower left")
plt.show()And this is the screenshot of the plot that is created in the notebook as seen using the onedork theme without customization.This definitely does not match the theme you have chosen. It turns out that in order to customize Matplotlib to match the theme you will need to add two additional lines of code at the top of your notebook.from jupyterthemes import jtplot
jtplot.style()Once you run the same chart code (with the two code lines from above added at the top of the notebook) you should see that the chart now matches the current style of the theme.That looks much better!You can actually change the graph style to match any theme that you want.from jupyterthemes import jtplot
jtplot.style(<theme_name>)If you do not give a parameter theme for the style function it will use the theme that is currently loaded in the notebook.Note unlike with theme setting there is no need to restart the notebook by changing the Matplotlib graphing style.Changing fontsJupyter Notebook themes do not only allow you to change themes but also do some additional customization regarding the fonts being used in the notebook.You can change the fonts when loading a theme with jt command and adding some additional parameters. You can customize…font used for code (-f) and its size (-fs),font used for notebook (-nf) and its size (-nfs) ,font used for text/markdown (-ft) and its size (-fts).SummaryIn this article, you have learned how to customize your standard Jupyter Notebook with Jupyter Notebook themes. We went through the details of the installation of the library and set up the themes including graph and font customization.You should be able to try it yourself now.Happy customizing!Originally published at aboutdatablog.com: Customize your Jupyter Notebooks with Jupyter Notebooks Themes, on January 20, 2021.
Machine Learning
Libraries
Analyzing Customer Conversions with Machine Learning and Decision Trees
Analyzing Customer Conversions with Machine Learning and Decision Trees
Visualization
Pandas
Libraries
Using Predictive Power Score to Pinpoint Non-linear Correlations
Using Predictive Power Score to Pinpoint Non-linear Correlations
Business
Libraries
Pandas
Using Python And Pandas Datareader to Analyze Financial Data
Finance and economics are becoming more and more interesting for all kinds of people, regardless of their career or profession. This is because we are all affected by economic data, or at least we are increasingly interested in being up-to-date, and we have a lot of information at hand.Every day billions of bytes of financial data are sent over the Internet. Whether it is the price of a share, an e-commerce transaction, or even information on a country's GDP. All this data, when properly organized and managed can be used to build some amazing and insightful software applications.
Pandas
Libraries
Using Pandas Profiling to Accelerate Our Exploratory Analysis
Pandas Profiling is a library that generates reports from a pandas DataFrame. The pandas df.describe() function that we normally use in Pandas is great but it is a bit basic for a more serious and detailed exploratory data analysis. pandas_profiling extends the pandas DataFrame with df.profile_report() for a quick data analysis.
Libraries
Python
Programming
Understanding Python's Collections Module
The Python collections module has different specialized data types that function as containers and can be used to replace the general purpose Python containers (`dict`, `tuple`, `list` and `set`). We will study the following parts of this module:- `ChainMap`- `defaultdict`- `deque`There is a submodule of collections called abc or Abstract Base Classes. These will not be covered in this post, let's start with the ChainMap container!ChainMapA ChainMap is a class that provides the ability to link multiple mappings together so that they end up as a single unit. If you look at the documentation, you will notice that it accepts `**maps*`, which means that a ChainMap will accept any number of mappings or dictionaries and convert them into a single view that you can update. Let's see an example so you can see how it works:from collections import ChainMap
car_parts = {'hood': 500, 'engine': 5000, 'front_door': 750}
car_options = {'A/C': 1000, 'Turbo': 2500, 'rollbar': 300}
car_accessories = {'cover': 100, 'hood_ornament': 150, 'seat_cover': 99}
car_pricing = ChainMap(car_accessories, car_options, car_parts)car_pricing
>> ChainMap({'cover': 100, 'hood_ornament': 150, 'seat_cover': 99}, {'A/C': 1000, 'Turbo': 2500, 'rollbar': 300}, {'hood': 500, 'engine': 5000, 'front_door': 750})
car_pricing['hood']
>> 500Here we import ChainMap from our collections module. Next we create three dictionaries. Then we create an instance of our ChainMap by passing it the three dictionaries we just created.Finally, we try to access one of the keys of our ChainMap. When we do this, the ChainMap will go through each map to see if that key exists and has a value. If it does, then the ChainMap will return the first value it finds that matches that key.This is especially useful if you want to set default values. Suppose we want to create an application that has some default values. The application will also know the operating system environment variables. If there is an environment variable that matches one of the keys we have as a default in our application, the environment will override our default value. In addition, let's assume that we can pass arguments to our application. These arguments take precedence over the environment and the defaults. This is one place where a ChainMap can really come in handy. Let's look at a simple example that is based on one from the Python documentation:Note: do not run this code from Jupyter Notebook, but from your favorite IDE and calling it from a terminal. this command `python chain_map.py -u daniel`import argparse
import os
from collections import ChainMap
def main():
app_defaults = {'username':'admin', 'password':'admin'}
parser = argparse.ArgumentParser()
parser.add_argument('-u', '--username')
parser.add_argument('-p', '--password')
args = parser.parse_args()
command_line_arguments = {key:value for key, value in vars(args).items() if value}
chain = ChainMap(command_line_arguments, os.environ, app_defaults)
print(chain['username'])
if __name__ == '__main__':
main()
os.environ['username'] = 'test'
main()➜ python python3 post.py -u daniel
daniel
daniel
Let's break this down a bit. Here we import the Python `argparse` module along with the `os` module. We also import `ChainMap`.Next we have a simple function that has some defaults. I have seen these defaults used for some popular routers. We then set up our argument parser and tell it how to handle certain command line options. You'll notice that argparse doesn't provide a way to get a dictionary object from its arguments, so we use a dict comprehension to extract what we need. The other interesting piece here is the use of Python's built-in vars. If you called it without a vars argument it would behave like Python's built-in locales. But if you pass it an object, then vars is the equivalent of the `__dict__` property of object. In other words, vars(args) is equal to `args.__dict__`. Finally we create our ChainMap by passing our command line arguments (if any), then the environment variables and finally the default values.At the end of the code, we try calling our function, then setting an environment variable and calling it again. Try it and you will see that it prints admin and then tests as expected. Now let's try calling the script with a command line argument:python chain_map.py -u danielWhen I run this on my machine, it returns daniel twice. This is because our command line argument overrides everything else. It doesn't matter what we set the environment to because our ChainMap will look at the command line arguments first before anything else. If you try it without the `-u daniel` it will run the actual arguments, in my case `"admin" "test"`.Now that you know how to use ChainMaps, we can move on to Counter!CounterThe collections module also provides us with a small tool that allows us to perform convenient and fast counting. This tool is called `Counter`. You can run it with most iterables. Let's try it with a stringfrom collections import Counter
Counter('superfluous')
>> Counter({'s': 2, 'u': 3, 'p': 1, 'e': 1, 'r': 1, 'f': 1, 'l': 1, 'o': 1})counter = Counter('superfluous')
counter['u']
>> 3In this example, we import `Counter` from `collections` and pass it a string. This returns a Counter object which is a subclass of the Python dictionary. We then run the same command but assign it to the counter variable so that we can access the dictionary more easily. In this case, we have seen that the letter `"u"` appears three times in the example string.The counter provides some methods that you may be interested in. For example, you can call elements which will get an iterator over the elements that are in the dictionary, but in an arbitrary order. This function can be considered as an "encoder", since the output in this case is an encoded version of the string.list(counter.elements())
>> ['s', 's', 'u', 'u', 'u', 'p', 'e', 'r', 'f', 'l', 'o']Another useful method is most_common. You can ask the Counter what are the most common elements by passing a number that represents what are the most recurring `"n"` elements:counter.most_common(2)
[('u', 3), ('s', 2)]Here we just asked our Counter which were the two most recurring items. As you can see, it produced a list of tuples that tells us that `"u"` occurred 3 times and `"s"` occurred twice.The other method I want to cover is the subtract method. The `subtract` method accepts an iterable or a mapping and uses that argument to subtract. It's a little easier to explain if you see some code:counter_one = Counter('superfluous')
counter_one
>> Counter({'s': 2, 'u': 3, 'p': 1, 'e': 1, 'r': 1, 'f': 1, 'l': 1, 'o': 1})
counter_two = Counter('super')
counter_one.subtract(counter_two)
counter_one
>> Counter({'s': 1, 'u': 2, 'p': 0, 'e': 0, 'r': 0, 'f': 1, 'l': 1, 'o': 1})So here we recreate our first counter and print it out so we know what's in it. Thus we create our second Counter object. Finally we subtract the second counter from the first. If you look closely at the output at the end, you will notice that the number of letters in five of the elements has been decreased by one.As I mentioned at the beginning of this section, you can use Counter against any iterable or mapping, so you don't have to use only strings. You can also pass tuples, dictionaries and lists to it.Try it on your own to see how it works with those other data types. Now we are ready to move on to `defaultdict`!`defaultdict`The collections module has a handy tool called `defaultdict`. The `defaultdict` is a subclass of the Python dict that accepts a `default_factory` as its main argument. The `default_factory` is usually a Python data type, such as int or a list, but you can also use a function or a lambda. Let's start by creating a regular Python dictionary that counts the number of times each word is used in a sentence:sentence = "The red for jumped over the fence and ran to the zoo for food"
words = sentence.split(' ')
words
>> ['The',
'red',
'for',
'jumped',
'over',
'the',
'fence',
'and',
'ran',
'to',
'the',
'zoo',
'for',
'food']
reg_dict = {}
for word in words:
if word in reg_dict:
reg_dict[word] += 1
else:
reg_dict[word] = 1
print(reg_dict)
>> {'The': 1, 'red': 1, 'for': 2, 'jumped': 1, 'over': 1, 'the': 2, 'fence': 1, 'and': 1, 'ran': 1, 'to': 1, 'zoo': 1, 'food': 1}Now let's try to do the same with defaultdict!from collections import defaultdict
sentence = "The red for jumped over the fence and ran to the zoo for food"
words = sentence.split(' ')
d = defaultdict(int)
for word in words:
d[word] += 1
print(d)
>> defaultdict(<class 'int'>, {'The': 1, 'red': 1, 'for': 2, 'jumped': 1, 'over': 1, 'the': 2, 'fence': 1, 'and': 1, 'ran': 1, 'to': 1, 'zoo': 1, 'food': 1})You will notice right away that the code is much simpler. The defaultdict will automatically assign zero as a value to any key it doesn't already have in it. We add one to make it make more sense and it will also increment if the word appears multiple times in the sentence.Now let's try using a Python list type as our `default_factory`. We will start with a regular dictionary, as before.my_list = [(1234, 100.23), (345, 10.45), (1234, 75.00), (345, 222.66), (678, 300.25), (1234, 35.67)]
reg_dict = {}
for acct_num, value in my_list:
if acct_num in reg_dict:
reg_dict[acct_num].append(value)
else:
reg_dict[acct_num] = [value]If you run this code, you should get output similar to the following:print(reg_dict)
>> {1234: [100.23, 75.0, 35.67], 345: [10.45, 222.66], 678: [300.25]}Now let's reimplement this code using defaultdict:from collections import defaultdict
my_list = [(1234, 100.23), (345, 10.45), (1234, 75.00), (345, 222.66), (678, 300.25), (1234, 35.67)]
d = defaultdict(list)
for acct_num, value in my_list:
d[acct_num].append(value)Again, this eliminates the if/else conditional logic and makes the code easier to follow. Here is the output of the above code:print(d)
>> defaultdict(<class 'list'>, {1234: [100.23, 75.0, 35.67], 345: [10.45, 222.66], 678: [300.25]})This is a very good thing! Let's try using a `lambda` also as our `default_factory`!from collections import defaultdict
animal = defaultdict(lambda: "Monkey")
animal
>> defaultdict(<function __main__.<lambda>()>, {})
animal['Sam'] = 'Tiger'
print (animal['Nick'])
>> Monkey
animal
>> defaultdict(<function __main__.<lambda>()>, {'Sam': 'Tiger', 'Nick': 'Monkey'})Here we create a `defaultdict` that will assign `Monkey` as default value to any key. The first key is set to `Tiger`, and the next key is not set. If you print the second key, you will see that it has 'Monkey' assigned to it. In case you haven't noticed yet, it is basically impossible to cause a KeyError as long as you set the `default_factory` to something that makes sense. The documentation mentions that if you set the `default_factory` to `None`, then you will receive a KeyError.Let's see how that works:from collections import defaultdict
x = defaultdict(None)
x['Mike']
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-30-d21c3702d01d> in <module>
1 from collections import defaultdict
2 x = defaultdict(None)
----> 3 x['Mike']
KeyError: 'Mike'In this case, we just created a `defaultdict` with an error. It can no longer assign a default value to our key, so it throws a `KeyError` instead. Of course, since it is a subclass of `dict`, we can simply set the key to some value and it will work. But that defeats the purpose of `defaultdict`.`deque`According to the Python documentation, deques "is a generalization of stacks and queues (stacks and queues)". It is pronounced `deck`, which is short for `double-ended queue`. They are a replacement container for the Python list. Deques are thread-safe and allow you to efficiently add and remove data from memory from either side of the deque. A list is optimized for fast fixed-length operations. Full details can be found in the Python documentation. A deque accepts a `maxlen` argument that sets the limits for the deque. Otherwise, the deque will grow to an arbitrary size. When a bounded deque is full, any new elements added will cause the same number of elements to come out the other end.As a general rule, if you need to add or remove elements quickly, use a deque. If you need quick random access use a list. Let's take a moment to see how a deque can be created and used.from collections import deque
import string
d = deque(string.ascii_lowercase)
for letter in d:
print(letter)
>> a
b
c
d
e
f
g
h
i
j
k
l
m
n
o
p
q
r
s
t
u
v
w
x
y
zHere we import the deque from our collections module and we also import the strings module. To create an instance of a deque, we need to pass it an iterable. In this case, we pass `string.ascii_lowerercase`, which returns a list of all the lowercase letters of the alphabet. Finally, we loop over our deque and print each element. Now let's look at some of the methods that deque has.d.append('bye')
d
>> deque(['a',
'b',
'c',
'd',
'e',
'f',
'g',
'h',
'i',
'j',
'k',
'l',
'm',
'n',
'o',
'p',
'q',
'r',
's',
't',
'u',
'v',
'w',
'x',
'y',
'z',
'bye'])
d.appendleft('hello')
d
>> deque(['hello',
'a',
'b',
'c',
'd',
'e',
'f',
'g',
'h',
'i',
'j',
'k',
'l',
'm',
'n',
'o',
'p',
'q',
'r',
's',
't',
'u',
'v',
'w',
'x',
'y',
'z',
'bye'])
d.rotate(1)
d
>> deque(['bye',
'hello',
'a',
'b',
'c',
'd',
'e',
'f',
'g',
'h',
'i',
'j',
'k',
'l',
'm',
'n',
'o',
'p',
'q',
'r',
's',
't',
'u',
'v',
'w',
'x',
'y',
'z'])Let's break this down a bit. First we add a chain to the right end of the deque. Then we add another string to the left side of the deque. Finally, we call `rotate` on our deque and pass it a one, which causes it to rotate once to the right. In other words, it makes an element rotate from the far right and in front. You can pass it a negative number to make the deque rotate to the left instead. Let's end this section with an example based on some of the Python documentationfrom collections import deque
def get_last(filename, n=5):
"""
Returns the last n lines from the file
"""
try:
with open(filename) as f:
return deque(f, n)
except OSError:
print("Error opening file: {}".format(filename))
raiseThis code works much like the Linux tail program does. Here we pass a `filename` to our script along with the `n` number of lines we want it to return. The deque is limited to whatever number we pass as `n`. This means that once the deque is full, when new lines are read and added to the deque, the oldest lines are pulled from the other end and discarded. I have also wrapped the file opening with a simple exception handler because it is very easy to pass a malformed path. This will catch files that do not exist.ConclusionWe've covered a lot of ground in this post. You learned how to use a defaultdict and a Count. We also learned about a subclass of the Python list, the deque. Finally, we saw how to use them to perform various activities. I hope you found each of these collections interesting. They may be of great use to you in your day-to-day work.
Deployment
Libraries
Flask
Machine Learning
Creating the Whole Machine Learning Pipeline with PyCaret
This tutorial covers the entire ML process, from data ingestion, pre-processing, model training, hyper-parameter fitting, predicting and storing the model for later use.We will complete all these steps in less than 10 commands that are naturally constructed and very intuitive to remember, such ascreate_model(),
tune_model(),
compare_models()
plot_model()
evaluate_model()
predict_model()
