Python Books You Must Read in 2020

Claire D
Oct 06, 2020

Contents Outline

Python Books You Must Read in 2020

Oct 06, 2020 11 minutes read

Python is one of thetop programming languages for a diverse range of tasks and domains. Python’s user-friendliness, high-level nature, and the emphasis on simplicity and enhanced code readability make it a favorable choice for many developers around the world. If that doesn’t sell Python to you, I’m sure that its exhaustive ecosystem of more than 255 thousand third-party packages will.

Features like these have skyrocketed the demand for Python everywhere, be it application development, Data Science, Artificial Intelligence, or any other industry. The goal behind this write-up is to round up some of the best Python books around, to help you gain the knowledge and confidence with this amazing programming language.

According to Stackoverflow, Python is the most preferred language which means that the majority of developers use python.

Python Books


Books are quite possibly one of the top sources of information about almost any topic, and in this section, we have gathered more than ten top books to help you familiarize and gain some practical knowledge with Python. Some of these books cover comprehensive knowledge about the programming language while some are excellent in giving you hands-on experience with it.

Regardless of your previous experience with Python or any programming language for that matter, we’re sure you will find some great handy tips from these books for your next project.

Disclaimer: Those affiliate links are for information purposes only.


1. Python Crash Course


Author: Eric Matthes

Publisher — No Starch Press

Difficulty Level: Beginners

Get Both Books here — Amazon, Amazon


Cover of the book “Python Crash Course”

As the name suggests, the author has written this book to act as a quick crash course for readers with little to no programming exposure. The author has made all the introductory concepts as easy as ABC for beginners so that they can start implementing their knowledge on fun projects. The introductory nature of this book also makes it a fitting choice for academics.

This two-part book covers the introduction to programming in its first part, whereas in the second part, it takes on a project-driven approach where the readers are encouraged to complete any or all three programming projects. The projects include coding a 2D game, creating a data visualization program, and the last, an online Learning Log for note-taking.

2. Learning Python


Author: Mark Lutz

Publisher — O’Reilly Media

Difficulty Level: Beginners

Get Book here — Amazon


Cover of the book “Learning Python”

In the Python way of thinking, explicit is better than implicit, and simple is better than complex.
― Mark Lutz

Learning Python covers all the fundamentals of the programming language and aims to be a one-stop solution for beginners alike who are in search of an in-depth introduction to Python. Also being a two-part book like the previous one, the author Mark Lutz has tried to create a solid foundation with Python in this part, while the other part focusses more on real-life examples and situations for better practical programming exposure.

The latest edition of the book covers Python v3.3 and all its latest improvements along with the older v2.7. On a side note, if you have zero proficiency with programming, it would be a good idea to supplement this book with additional introductory references.


3. Python Tricks: A Buffet of Awesome Python Features


Author: Dan Bader

Publisher — Dan Bader (dbader.org)

Difficulty Level: Beginners

Get Book here — Amazon


Cover of the book “Python Tricks: A Buffet of Awesome Python Features”

“There should be one — and preferably only one — obvious way to do it.”
― Dan Bader

As the title suggests, Python Tricks brings together a collection of convenient features, tips, and tricks to make you efficient with Python.

In the words of the author — “What started out as a fun twitter experiment, turned into a series of noteworthy and useful tricks accompanied by a clear code example, that helped hundreds of Python developers understand the idea behind the various aspects of Python.”

The book covers a large collection of tricks from a variety of topics in Python, and they’re presented in a well-explained style, but to make full use of this book, you’d still need a strong foundation in Python.

4. Learn Python the Hard Way


Author: Zed Shaw

Publisher — Addison-Wesley

Difficulty Level: Beginners

Get Book here — Amazon


Cover of the book “Learn Python the Hard Way”

“Just take it slow and do not hurt your brain.”

If you’re intimidated by the “Hard” in the title, don’t be. That’s just the author’s way of using instructions to make you thoroughly go through the chapters and practice what you’ve learned.

Putting the title aside, the author has done an amazing job presenting the fundamental concepts of Python at a more beginner-friendly pace to prepare you for complex topics. The book also includes plenty of instructional videos and exercises to enhance your knowledge of Python. In case you’re wondering, yes, the book has been updated with a newer edition that supports Python 3.

5. Automating Boring Stuff with Python


Author: Al Sweigart

Publisher — No Starch Press

Difficulty Level: Beginners

Get Book here — Amazon


Cover of the book “Automating Boring Stuff with Python”

According to the author-

Don’t spend your time doing work a well-trained monkey could do. Even if you’ve never written a line of code, you can make your computer do the grunt work. Learn how in Automate the Boring Stuff with Python.

The title says it all. Every now and then, you must’ve come across a boring or a repetitive task that makes you say, “not this again.” It is these moments that this book intends to eliminate. The author has covered the necessary basics of the programming language in this book to help you create some nifty snippets of Python that can automate a simple but boring task to be done in seconds instead of hours.

While the book does wonders for anyone wanting to get the boring tasks out of the way quickly, sans the programming background, it does not comprehensively cover each aspect of Python’s. Good enough for creating handy throwaway code but not enough for a thorough introduction.


6. Python for Data Analysis


Author: Wes McKinney

Publisher — O′Reilly

Difficulty Level: Intermediate

Get Book here — Amazon


Cover of the book “Python for Data Analysis”

Python can be used for a variety of tasks, and one of them is data analysis. If you constantly find yourself occupied with analyzing and manipulating structured data or are simply keen on learning about how efficient Python can be for data analysis tasks, you might find this book useful.

The author has explained the fundamentals of working with data in a very comprehensive manner while also touching upon the topic of scientific computing. Python for Data Analysis also covers some of the most popular libraries for data analysis, such as NumPy, pandas, matplotlib, IPython, and SciPy.

“Act without doing; work without effort. Think of the small as large and the few as many. Confront the difficult while it is still easy; accomplish the great task by a series of small acts. — Laozi”
― Wes McKinney


7. Introduction to Machine Learning with Python


Author: Andreas C. Müller and Sarah Guido

Publisher — O′Reilly

Difficulty Level: Intermediate

Get Book here — Amazon


Cover of the book “Introduction to Machine Learning with Python”

The rate at which Machine Learning is advancing is fascinating. To be able to make the most out of this technology, Python is among the top choices for a glue language. Targeted towards aspiring Machine Learning professionals in search of solutions to real-world machine learning problems, this introductory book requires zero prior experience with Machine Learning.

Instead of diving into the mathematics behind the algorithms and models being used throughout, the book takes a gentler approach and explains the background and their importance. Though it does require some knowledge of using Python to implement the vast collection of algorithms and models covered by the libraries followed in the book, such asscikit-learn, NumPy, and matplotlib.


8. Python Data Science Handbook


Author: Jake VanderPlas

Publisher — O’Reilly Media

Difficulty Level: Intermediate

Get Book here — Amazon


Cover of the book “Python Data Science Handbook”

Data Science is becoming more of an imperative skill in various domains lately as the benefits it offers is invaluable. The author of this handbook has put more emphasis on learning Data Science as a skill than a new domain of knowledge, as it can prove to be advantageous while applying the skill to a problem in hand.

The contents of the book have been structured into five different libraries provided by Python for extensive coverage, i.e, IPython, NumPy, Pandas, Matplotlib, Scikit-Learn. The handbook does require a certain degree of proficiency in Python to follow the book as intended by the author, and would not make for a suitable choice for beginners.

9. Head First Python: A Brain-Friendly Guide


Author: Paul Barry

Publisher — O’Reilly Media

Difficulty Level: Intermediate

Get Book here — Amazon


Cover of the book “Head First Python: A Brain-Friendly Guide”

“code is read more than it’s written. This”
― Paul Barry

Although this book doesn’t cover the programming language and its entirety in detail, it still manages to teach you Python in a more practical and fun way. What separates this book from others is the fun and casual style the author has used to build a connection with the readers, and more importantly, the comical use of pictures to keep the learning process intuitive.

For those of you who have just started learning Python, or any other programming language for that matter, you might have a rough time getting your head around the topics covered. If you do know the basics of programming, hop on.

10. Fluent Python


Author: Luciano Ramalho

Publisher — O’Reilly Media

Difficulty Level: Experts

Get Book here — Amazon


Cover of the book “Fluent Python”

Python can be a very versatile and powerful programming language when used efficiently, and this very motive is the sole driving factor for the book Fluent Python. It is clear that learning Python and achieving fluency in Python are both very different things. Most developers will often achieve what they want with Python anyways but at the cost of its full potential.

The author has emphasized on highlighting some of the less utilized features and techniques to make it possible to get the most out of Python. If you have recently started learning Python, this might not be the right book for you, as you might find it hard to follow.

11. Effective Python


Author: Brett Slatkin

Publisher — Addison-Wesley

Difficulty Level: Experts

Get Book here — Amazon


Cover of the book “ Effective Python”

Python is a programming language that puts a high emphasis on creating clear and extremely readable code, but there can still be some situations where that isn’t achieved. This is where the book Effective Python comes into the picture. The author has covered some common mistakes and provided invaluable insights and practices on how to avoid them in the first place, to write cleaner, reusable, and more effective Python code. If you’re constantly struggling to find ways to optimize your code, this could be the right book for you.


12. Python Cookbook


Author: David Beazley & Brian K. Jones

Publisher — O’Reilly Media

Difficulty Level: Experts

Get Book here — Amazon


Cover of the book “Python Cookbook”

Python has an excellent community, and this book takes its inspiration from the many unique challenges faced by the community, which are referred to as recipes in the Python Cookbook. The recipes come with relevant examples and detailed background studies on the problems from some of the most insightful members of the community.

To make it absolutely clear, the author has targeted this book towards experienced Python developers looking to strengthen their understanding of the various modern techniques in Python. Beginners are suggested to pick up something that covers the introductory parts of Python before starting with the Python Cookbook.



More Python Books to Read —

  • Python Pocket Reference
  • Python Machine Learning
  • Deep Learning with Python
  • Python Programming: An Introduction to Computer Science
  • Natural Language Processing with Python
  • Python in a Nutshell
  • Think Python: How to Think Like a Computer Scientist
  • Django for Beginners: Build websites with Python and Django



Conclusion


Python is an elegant and powerful programming language that can do wonders if utilized correctly. The books covered in this write-up should provide you sufficient knowledge to get you started with Python along with some additional tips and tricks to write a clear and optimized code that works beautifully. To tie things up here, if you are keen on learning Python from scratch or would simply like a brush-up, we highly recommend you go through these books.

Note: To eliminate problems of different kinds, I want to alert you to the fact this article represent just my personal opinion I want to share, and you possess every right to disagree with it.

Related Posts

Why Decorators In Python Are Pure Genius?

3 Python Tricks That Will Improve Your Code

Top 10 Python Extensions for Visual Studio Code

Categories

Data Science Machine Learning Deep learning Flask Deployment Big Data Business Python SQL Programming Libraries Jupyter Numpy News Pandas Matplotlib Visualization Datathons

Join Competition

Financial Inclusion in LatAm

Prediction of Online Shoppers Purchasing Intention

Most Related Articles

Python
Programming
Why Decorators In Python Are Pure Genius?

Analyze, test, and re-use your code with little more than an @ symbolIf there’s one thing that makes Python incredibly successful, that would be its readability. Everything else hinges on that: if code is unreadable, it’s hard to maintain. It’s also not beginner-friendly then — a novice getting boggled by unreadable code won’t attempt writing its own one day.Python was already readable and beginner-friendly before decorators came around. But as the language started getting used for more and more things, Python developers felt the need for more and more features, without cluttering the landscape and making code unreadable.Decorators are a prime-time example of a perfectly implemented feature. It does take a while to wrap your head around, but it’s worth it. As you start using them, you’ll notice how they don’t overcomplicate things and make your code neat and snazzy.Before anything else: higher-order functionsIn a nutshell, decorators are a neat way to handle higher-order functions. So let’s look at those first!Functions returning functionsSay you have one function, greet() — it greets whatever object you pass it. And let’s say you have another function, simon() — it inserts “Simon” wherever appropriate. How can we combine the two? Think about it a minute before you look below.def greet(name): return f"Hello, {name}!" def simon(func): return func("Simon") simon(greet)The output is 'Hello, Simon!'. Hope that makes sense to ya!Of course we could have just called greet("Simon"). However, the whole point is that we might want to put “Simon” into many different functions. And if we don’t use “Simon” but something more complicated, we can save a whole lot of lines of code by packing it into a function like simon().Functions inside other functionsWe can also define functions inside other functions. That’s important because decorators will do that, too! Without decorators it looks like this:def respect(maybe): def congrats(): return "Congrats, bro!" def insult(): return "You're silly!" if maybe == "yes": return congrats else: return insultThe function respect() returns a function; respect("yes") returns the congrats function, respect("brother") (or some other argument instead of "brother") returns the insult function. To call the functions, enter respect("yes")() and respect("brother")(), just like a normal function.Also read: Python Books You Must Read in 2020Got it? Then you’re all set for decorators!Code is beautifully nerdy. Image by author.The ABC of Python decoratorsFunctions with an @ symbolLet’s try a combination of the two previous concepts: a function that takes another function and defines a function. Sounds mind-boggling? Consider this:def startstop(func): def wrapper(): print("Starting...") func() print("Finished!") return wrapper def roll(): print("Rolling on the floor laughing XD") roll = startstop(roll)The last line ensures that we don’t need to call startstop(roll)() anymore; roll() will suffice. Do you know what the output of that call is? Try it yourself if you’re unsure!Now, as a very good alternative, we could insert this right after defining startstop():@startstop def roll(): print("Rolling on the floor laughing XD")This does the same, but glues roll() to startstop() at the onset.Added flexibilityWhy is that useful? Doesn’t that consume exactly as many lines of code as before?In this case, yes. But once you’re dealing with slightly more complicated stuff, it gets really useful. For once, you can move all decorators (i.e. the def startstop() part above) into its own module. That is, you write them into a file called decorators.py and write something like this into your main file:from decorators import startstop @startstop def roll(): print("Rolling on the floor laughing XD")In principle, you can do that without using decorators. But this way it makes life easier because you don’t have to deal with nested functions and endless bracket-counting anymore.You can also nest decorators:from decorators import startstop, exectime @exectime @startstop def roll(): print("Rolling on the floor laughing XD")Note that we haven’t defined exectime() yet, but you’ll see it in the next section. It’s a function that can measure how long a process takes in Python.This nesting would be equivalent to a line like this:roll = exectime(startstop(roll))Bracket counting is starting! Imagine you had five or six of those functions nested inside each other. Wouldn’t the decorator notation be much easier to read than this nested mess?You can even use decorators on functions that accept arguments. Now imagine a few arguments in the line above and your chaos would be complete. Decorators make it neat and tidy.Also read: How to Get a Job With PythonFinally, you can even add arguments to your decorators — like @mydecorator(argument). Yeah, you can do all of this without decorators. But then I wish you a lot of fun understanding your decorator-free code when you re-read it in three weeks…Decorators make everything easier. Image by author.Applications: where decorators cut the creamNow that I’ve hopefully convinced you that decorators make your life three times easier, let’s look at some classic examples where decorators are basically indispensable.Measuring execution timeLet’s say we have a function called waste time() and we want to know how long it takes. Well, just use a decorator!import time def measuretime(func): def wrapper(): starttime = time.perf_counter() func() endtime = time.perf_counter() print(f"Time needed: {endtime - starttime} seconds") return wrapper @measuretime def wastetime(): sum([i**2 for i in range(1000000)]) wastetime()A dozen lines of code and we’re done! Plus, you can use measuretime() on as many functions as you want.Slowing code downSometimes you don’t want to execute code immediately but wait a while. That’s where a slow-down decorator comes in handy:import time def sleep(func): def wrapper(): time.sleep(300) return func() return wrapper @sleep def wakeup(): print("Get up! Your break is over.") wakeup()Calling wakeup() makes lets you take a 5-minute break, after which your console reminds you to get back to work.Also read: Building A Linear Regression Model With Python To Predict Retail Customer SpendingTesting and debuggingSay you have a whole lot of different functions that you call at different stages, and you’re losing the overview over what’s being called when. With a simple decorator for every function definition, you can bring more clarity. Like so:def debug(func): def wrapper(): print(f"Calling {func.__name__}") return wrapper @debug def scare(): print("Boo!") scare()There is an a lot more elaborate example here. Note, though, that to understand that example, you’ll have to check how to decorate functions with arguments. Still, it’s worth the read!Reusing codeThis kinda goes without saying. If you’ve defined a function decorator(), you can just sprinkle @decorator everywhere in your code. To be honest, I don’t think it gets any simpler than that!Handling loginsIf you have functionalities that should only be accessed if a user is logged in, that’s also fairly easy with decorators. I’ll refer you to the full example for reference, but the principle is quite simple: first you define a function like login_required(). Before any function definition that needs logging in, you pop @login_required. Simple enough, I’d say.Syntactic sugar — or why Python is so sweetIt’s not like I’m not critical of Python or not using alternative languages where it’s appropriate. But there’s a big allure to Python: it’s so easy to digest, even when you’re not a computer scientist by training and just want to make things work.If C++ is an orange, then Python is a pineapple: similarly nutritious, but three times sweeter. Decorators are just one factor in the mix.But I hope you’ve come to see why it’s such a big sweet-factor. Syntactic sugar to add some pleasure to your life! Without health risks, except for having your eyes glued on a screen.I wish you lots of sweet code!Also Read: How to Use Python Datetimes Correctly?

Data Science
Programming
6 Advanced Statistical Concepts in Data Science

The article contains some of the most commonly used advanced statistical concepts along with their Python implementation.In my previous articles Beginners Guide to Statistics in Data Science and The Inferential Statistics Data Scientists Should Know we have talked about almost all the basics(Descriptive and Inferential) of statistics which are commonly used in understanding and working with any data science case study. In this article, lets go a little beyond and talk about some advance concepts which are not part of the buzz.Concept #1 - Q-Q(quantile-quantile) PlotsBefore understanding QQ plots first understand what is a Quantile?A quantile defines a particular part of a data set, i.e. a quantile determines how many values in a distribution are above or below a certain limit. Special quantiles are the quartile (quarter), the quintile (fifth), and percentiles (hundredth).An example:If we divide a distribution into four equal portions, we will speak of four quartiles. The first quartile includes all values that are smaller than a quarter of all values. In a graphical representation, it corresponds to 25% of the total area of distribution. The two lower quartiles comprise 50% of all distribution values. The interquartile range between the first and third quartile equals the range in which 50% of all values lie that are distributed around the mean. In Statistics, A Q-Q(quantile-quantile) plot is a scatterplot created by plotting two sets of quantiles against one another. If both sets of quantiles came from the same distribution, we should see the points forming a line that’s roughly straight(y=x).Q-Q plotFor example, the median is a quantile where 50% of the data fall below that point and 50% lie above it. The purpose of Q Q plots is to find out if two sets of data come from the same distribution. A 45-degree angle is plotted on the Q Q plot; if the two data sets come from a common distribution, the points will fall on that reference line.It’s very important for you to know whether the distribution is normal or not so as to apply various statistical measures on the data and interpret it in much more human-understandable visualization and their Q-Q plot comes into the picture. The most fundamental question answered by the Q-Q plot is if the curve is Normally Distributed or not.Normally distributed, but why?The Q-Q plots are used to find the type of distribution for a random variable whether it is a Gaussian Distribution, Uniform Distribution, Exponential Distribution, or even Pareto Distribution, etc. You can tell the type of distribution using the power of the Q-Q plot just by looking at the plot. In general, we are talking about Normal distributions only because we have a very beautiful concept of the 68–95–99.7 rule which perfectly fits into the normal distribution So we know how much of the data lies in the range of the first standard deviation, second standard deviation and third standard deviation from the mean. So knowing if a distribution is Normal opens up new doors for us to experiment with  Types of Q-Q plots. Source Skewed Q-Q plotsQ-Q plots can find skewness(measure of asymmetry) of the distribution. If the bottom end of the Q-Q plot deviates from the straight line but the upper end is not, then the distribution is Left skewed(Negatively skewed).Now if upper end of the Q-Q plot deviates from the staright line and the lower is not, then the distribution is Right skewed(Positively skewed).Tailed Q-Q plotsQ-Q plots can find Kurtosis(measure of tailedness) of the distribution.The distribution with the fat tail will have both the ends of the Q-Q plot to deviate from the straight line and its centre follows the line, where as a thin tailed distribution will term Q-Q plot with very less or negligible deviation at the ends thus making it a perfect fit for normal distribution.Q-Q plots in Python(Source)Suppose we have the following dataset of 100 values:import numpy as np #create dataset with 100 values that follow a normal distribution np.random.seed(0) data = np.random.normal(0,1, 1000) #view first 10 values data[:10] array([ 1.76405235, 0.40015721, 0.97873798, 2.2408932 , 1.86755799, -0.97727788, 0.95008842, -0.15135721, -0.10321885, 0.4105985 ])To create a Q-Q plot for this dataset, we can use the qqplot() function from the statsmodels library:import statsmodels.api as sm import matplotlib.pyplot as plt #create Q-Q plot with 45-degree line added to plot fig = sm.qqplot(data, line='45') plt.show()In a Q-Q plot, the x-axis displays the theoretical quantiles. This means it doesn’t show your actual data, but instead, it represents where your data would be if it were normally distributed.The y-axis displays your actual data. This means that if the data values fall along a roughly straight line at a 45-degree angle, then the data is normally distributed.We can see in our Q-Q plot above that the data values tend to closely follow the 45-degree, which means the data is likely normally distributed. This shouldn’t be surprising since we generated the 100 data values by using the numpy.random.normal() function.Consider instead if we generated a dataset of 100 uniformly distributed values and created a Q-Q plot for that dataset:#create dataset of 100 uniformally distributed values data = np.random.uniform(0,1, 1000) #generate Q-Q plot for the dataset fig = sm.qqplot(data, line='45') plt.show()The data values clearly do not follow the red 45-degree line, which is an indication that they do not follow a normal distribution.Concept #2- Chebyshev's InequalityIn probability, Chebyshev’s Inequality, also known as “Bienayme-Chebyshev” Inequality guarantees that, for a wide class of probability distributions, only a definite fraction of values will be found within a specific distance from the mean of a distribution.Source: https://www.thoughtco.com/chebyshevs-inequality-3126547 Chebyshev’s inequality is similar to The Empirical rule(68-95-99.7); however, the latter rule only applies to normal distributions. Chebyshev’s inequality is broader; it can be applied to any distribution so long as the distribution includes a defined variance and mean.So Chebyshev’s inequality says that at least (1-1/k^2) of data from a sample must fall within K standard deviations from the mean (or equivalently, no more than 1/k^2 of the distribution’s values can be more than k standard deviations away from the mean).Where K --> Positive real numberIf the data is not normally distributed then different amounts of data could be in one standard deviation. Chebyshev’s inequality provides a way to know what fraction of data falls within K standard deviations from the mean for any data distribution.Also read: 22 Statistics Questions to Prepare for Data Science InterviewsCredits: https://calcworkshop.com/joint-probability-distribution/chebyshev-inequality/ Chebyshev’s inequality is of great value because it can be applied to any probability distribution in which the mean and variance are provided.Let us consider an example, Assume 1,000 contestants show up for a job interview, but there are only 70 positions available. In order to select the finest 70 contestants amongst the total contestants, the proprietor gives tests to judge their potential. The mean score on the test is 60, with a standard deviation of 6. If an applicant scores an 84, can they presume that they are getting the job?The results show that about 63 people scored above a 60, so with 70 positions available, a contestant who scores an 84 can be assured they got the job.Chebyshev's Inequality in Python(Source) Create a population of 1,000,000 values, I use a gamma distribution(also works with other distributions) with shape = 2 and scale = 2.import numpy as np import random import matplotlib.pyplot as plt #create a population with a gamma distribution shape, scale = 2., 2. #mean=4, std=2*sqrt(2) mu = shape*scale #mean and standard deviation sigma = scale*np.sqrt(shape) s = np.random.gamma(shape, scale, 1000000)Now sample 10,000 values from the population.#sample 10000 values rs = random.choices(s, k=10000)Count the sample that has a distance from the expected value larger than k standard deviation and use the count to calculate the probabilities. I want to depict a trend of probabilities when k is increasing, so I use a range of k from 0.1 to 3.#set k ks = [0.1,0.5,1.0,1.5,2.0,2.5,3.0] #probability list probs = [] #for each k for k in ks: #start count c = 0 for i in rs: # count if far from mean in k standard deviation if abs(i - mu) > k * sigma : c += 1 probs.append(c/10000)Plot the results:plot = plt.figure(figsize=(20,10)) #plot each probability plt.xlabel('K') plt.ylabel('probability') plt.plot(ks,probs, marker='o') plot.show() #print each probability print("Probability of a sample far from mean more than k standard deviation:") for i, prob in enumerate(probs): print("k:" + str(ks[i]) + ", probability: " \ + str(prob)[0:5] + \ " | in theory, probability should less than: " \ + str(1/ks[i]**2)[0:5])From the above plot and result, we can see that as the k increases, the probability is decreasing, and the probability of each k follows the inequality. Moreover, only the case that k is larger than 1 is useful. If k is less than 1, the right side of the inequality is larger than 1 which is not useful because the probability cannot be larger than 1.Concept #3- Log-Normal DistributionIn probability theory, a Log-normal distribution also known as Galton's distribution is a continuous probability distribution of a random variable whose logarithm is normally distributed.Thus, if the random variable X is log-normally distributed, then Y = ln(X) has a normal distribution. Equivalently, if Y has a normal distribution, then the exponential function of Y i.e, X = exp(Y), has a log-normal distribution. Skewed distributions with low mean and high variance and all positive values fit under this type of distribution. A random variable that is log-normally distributed takes only positive real values. The general formula for the probability density function of the lognormal distribution is:The location and scale parameters are equivalent to the mean and standard deviation of the logarithm of the random variable.The shape of Lognormal distribution is defined by 3 parameters:σis the shape parameter, (and is the standard deviation of the log of the distribution)θ or μ is the location parameter (and is the mean of the distribution)m is the scale parameter (and is also the median of the distribution)The location and scale parameters are equivalent to the mean and standard deviation of the logarithm of the random variable as explained above.If x = θ, then f(x) = 0. The case where θ = 0 and m = 1 is called the standard lognormal distribution. The case where θ equals zero is called the 2-parameter lognormal distribution.The following graph illustrates the effect of the location(μ) and scale(σ) parameter on the probability density function of the lognormal distribution: Source: https://www.sciencedirect.com/topics/mathematics/lognormal-distribution Log-Normal Distribution in Python(Source)Let us consider an example to generate random numbers from a log-normal distribution with μ=1 and σ=0.5 using scipy.stats.lognorm function.import numpy as np import matplotlib.pyplot as plt from scipy.stats import lognorm np.random.seed(42) data = lognorm.rvs(s=0.5, loc=1, scale=1000, size=1000) plt.figure(figsize=(10,6)) ax = plt.subplot(111) plt.title('Generate wrandom numbers from a Log-normal distribution') ax.hist(data, bins=np.logspace(0,5,200), density=True) ax.set_xscale("log") shape,loc,scale = lognorm.fit(data) x = np.logspace(0, 5, 200) pdf = lognorm.pdf(x, shape, loc, scale) ax.plot(x, pdf, 'y') plt.show()Concept #4- Power Law distributionIn statistics, a Power Law is a functional relationship between two quantities, where a relative change in one quantity results in a proportional relative change in the other quantity, independent of the initial size of those quantities: one quantity varies as a power of another. For instance, considering the area of a square in terms of the length of its side, if the length is doubled, the area is multiplied by a factor of four.A power law distribution has the form Y = k Xα, where:X and Y are variables of interest,α is the law’s exponent,k is a constant.Source: https://en.wikipedia.org/wiki/Power_law Power-law distribution is just one of many probability distributions, but it is considered a valuable tool to assess uncertainty issues that normal distribution cannot handle when they occur at a certain probability.Many processes have been found to follow power laws over substantial ranges of values. From the distribution in incomes, size of meteoroids, earthquake magnitudes, the spectral density of weight matrices in deep neural networks, word usage, number of neighbors in various networks, etc. (Note: The power law here is a continuous distribution. The last two examples are discrete, but on a large scale can be modeled as if continuous).Also read: Statistical Measures of Central TendencyPower-law distribution in Python(Source) Let us plot the Pareto distribution which is one form of a power-law probability distribution. Pareto distribution is sometimes known as the Pareto Principle or ‘80–20’ rule, as the rule states that 80% of society’s wealth is held by 20% of its population. Pareto distribution is not a law of nature, but an observation. It is useful in many real-world problems. It is a skewed heavily tailed distribution.import numpy as np import matplotlib.pyplot as plt from scipy.stats import pareto x_m = 1 #scale alpha = [1, 2, 3] #list of values of shape parameters plt.figure(figsize=(10,6)) samples = np.linspace(start=0, stop=5, num=1000) for a in alpha: output = np.array([pareto.pdf(x=samples, b=a, loc=0, scale=x_m)]) plt.plot(samples, output.T, label='alpha {0}' .format(a)) plt.xlabel('samples', fontsize=15) plt.ylabel('PDF', fontsize=15) plt.title('Probability Density function', fontsize=15) plt.legend(loc='best') plt.show()Concept #5- Box cox transformationThe Box-Cox transformation transforms our data so that it closely resembles a normal distribution.The one-parameter Box-Cox transformations are defined as In many statistical techniques, we assume that the errors are normally distributed. This assumption allows us to construct confidence intervals and conduct hypothesis tests. By transforming your target variable, we can (hopefully) normalize our errors (if they are not already normal).Additionally, transforming our variables can improve the predictive power of our models because transformations can cut away white noise.Original distribution(Left) and near-normal distribution after applying Box cox transformation. Source  At the core of the Box-Cox transformation is an exponent, lambda (λ), which varies from -5 to 5. All values of λ are considered and the optimal value for your data is selected; The “optimal value” is the one that results in the best approximation of a normal distribution curve. The one-parameter Box-Cox transformations are defined as:and the two-parameter Box-Cox transformations as:Moreover, the one-parameter Box-Cox transformation holds for y > 0, i.e. only for positive values and two-parameter Box-Cox transformation for y > -λ, i.e. negative values. The parameter λ is estimated using the profile likelihood function and using goodness-of-fit tests.If we talk about some drawbacks of Box-cox transformation, then if interpretation is what you want to do, then Box-cox is not recommended. Because if λ is some non-zero number, then the transformed target variable may be more difficult to interpret than if we simply applied a log transform.A second stumbling block is that the Box-Cox transformation usually gives the median of the forecast distribution when we revert the transformed data to its original scale. Occasionally, we want the mean and not the median.Box-Cox transformation in Python(Source)SciPy’s stats package provides a function called boxcox for performing box-cox power transformation that takes in original non-normal data as input and returns fitted data along with the lambda value that was used to fit the non-normal distribution to normal distribution.#load necessary packages import numpy as np from scipy.stats import boxcox import seaborn as sns #make this example reproducible np.random.seed(0) #generate dataset data = np.random.exponential(size=1000) fig, ax = plt.subplots(1, 2) #plot the distribution of data values sns.distplot(data, hist=False, kde=True, kde_kws = {'shade': True, 'linewidth': 2}, label = "Non-Normal", color ="red", ax = ax[0]) #perform Box-Cox transformation on original data transformed_data, best_lambda = boxcox(data) sns.distplot(transformed_data, hist = False, kde = True, kde_kws = {'shade': True, 'linewidth': 2}, label = "Normal", color ="red", ax = ax[1]) #adding legends to the subplots plt.legend(loc = "upper right") #rescaling the subplots fig.set_figheight(5) fig.set_figwidth(10) #display optimal lambda value print(f"Lambda value used for Transformation: {best_lambda}") Concept #6- Poisson distributionIn probability theory and statistics, the Poisson distribution is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time or space if these events occur with a known constant mean rate and independently of the time since the last event.In very simple terms, A Poisson distribution can be used to estimate how likely it is that something will happen "X" number of times. Some examples of Poisson processes are customers calling a help center, radioactive decay in atoms, visitors to a website, photons arriving at a space telescope, and movements in a stock price. Poisson processes are usually associated with time, but they do not have to be. The Formula for the Poisson Distribution Is:Where:e is Euler's number (e = 2.71828...)k is the number of occurrencesk! is the factorial of kλ is equal to the expected value of kwhen that is also equal to its varianceLambda(λ) can be thought of as the expected number of events in the interval. As we change the rate parameter, λ, we change the probability of seeing different numbers of events in one interval. The below graph is the probability mass function of the Poisson distribution showing the probability of a number of events occurring in an interval with different rate parameters.   Probability Mass function for Poisson Distribution with varying rate parameters.Source The Poisson distribution is also commonly used to model financial count data where the tally is small and is often zero. For one example, in finance, it can be used to model the number of trades that a typical investor will make in a given day, which can be 0 (often), or 1, or 2, etc.As another example, this model can be used to predict the number of "shocks" to the market that will occur in a given time period, say over a decade.Poisson distribution in Pythonfrom numpy import random import matplotlib.pyplot as plt import seaborn as sns lam_list = [1, 4, 9] #list of Lambda values plt.figure(figsize=(10,6)) samples = np.linspace(start=0, stop=5, num=1000) for lam in lam_list: sns.distplot(random.poisson(lam=lam, size=10), hist=False, label='lambda {0}'.format(lam)) plt.xlabel('Poisson Distribution', fontsize=15) plt.ylabel('Frequency', fontsize=15) plt.legend(loc='best') plt.show()As λ becomes bigger, the graph looks more like a normal distribution.I hope you have enjoyed reading this article, If you have any questions or suggestions, please leave a comment. Also read: False Positives vs. False NegativesFeel free to connect me on LinkedIn for any query.Thanks for reading!!!Referenceshttps://calcworkshop.com/joint-probability-distribution/chebyshev-inequality/ https://corporatefinanceinstitute.com/resources/knowledge/data-analysis/chebyshevs-inequality/ https://www.itl.nist.gov/div898/handbook/eda/section3/eda3669.htm https://www.statology.org/q-q-plot-python/ https://gist.github.com/chaipi-chaya/9eb72978dbbfd7fa4057b493cf6a32e7 https://stackoverflow.com/a/41968334/7175247 

Python
3 Python Tricks That Will Improve Your Code

I’ve been coding in Python for more than 10 years. There was a time when I thought I knew it all, which was a clear sign I was getting complacent.Then I decided to do a bit of research about Python improvements. Those 3.6, 3.7, 3.8 Python versions aren’t there for nothing, right?After going through release notes, I found these neat tricks that I would like to share with you.By reading this article, you’ll learn:How to format big integers more clearlyA better way to work with file pathsThe proper way of string formating1. Formatting big integersUsing Underscores in Numeric Literals. Image by Roman OracFrom Python 3.6 (and onwards) you can use underscores to make numbers easier to read. See PEP 515 for more details.Let’s look at an example:a = 1000000000 # Is variable a billion or 100 millions? # Let's use underscores to make it easier to read a = 1_000_000_000 # You can group numbers as you like b = 1_0_9_0It also works with hexadecimal addresses and grouping bits.# grouping hexadecimal addresses by words addr = 0xCAFE_F00D # grouping bits into nibbles in a binary literal flags = 0b_0011_1111_0100_1110Also read: Why Decorators In Python Are Pure Genius?2. PathlibPhoto by Alice Donovan Rouse on UnsplashWorking with paths can be challenging especially if your code needs to run on multiple operating systems.Luckily for us, Python standard library has pathllib.Let’s look at an example:from pathlib import Path path = Path("some_folder") print(path) # output: some_folder # We can add more subfolders in a readable way path = path / "sub_folder" / "sub_sub_folder" print(path) # output: some_folder/sub_folder/sub_sub_folder # make path absolute print(path.resolve()) # output: /Users/r.orac/some_folder/sub_folter/sub_sub_folderDespite using / sign as the separator to concatenate paths, pathlib makes it operating system agnostic, meaning it will also work on Windows OS.Also read: Using Python And Pandas Datareader to Analyze Financial Data3. Simplify string formattingf-string formatting in Python. Image by Roman OracI’m used to using old school string formatting in Python:person = 'Roman' exercise = 0 print("%d-times %s exercised during corona epidemic" % (exercise, person)) # output # 0-times Roman exercised during corona epidemicUntil recently, I didn’t know there is a better (more modern) way of string formatting in Python.In Python 3.6, PEP 498 introduces Literal String Interpolation, which simplifies the string formatting.We can rewrite the example above to:person = 'roman' exercise = 0 print(f"{exercise}-times {person} exercised during corona epidemic") # output # 0-times Roman exercised during corona epidemicA string prefixed with f is known as fstring.fstrings even support math operations:print(f"{exercise+1}-times {person} exercised during corona epidemic") # Output # '1-times roman exercised during corona epidemic'But I didn’t exercise during the corona epidemic so adding +1 in the fstring would simply be a lie 😂What about formatting float values?f = 0.333333 print(f"this is f={f:.2f} rounded to 2 decimals") # Output this is f=0.33 rounded to 2 decimalsAlso read: How to Use Python Datetimes Correctly?ConclusionThese Python tricks will make your code more concise and will make coding more enjoyable.Many Python developers don’t know about these tricks — you’re not one of them anymore.Before you goFollow me on Twitter, where I regularly tweet about Data Science and Machine Learning.Also read: Understanding Python's Collections Module

Python
Top 10 Python Extensions for Visual Studio Code

In this new post we want to talk about the most useful Python extensions for Visual Studio Code. Visual Studio Code is an integrated development environment created by Microsoft for Windows, Linux and macOS. Among its features are debugging, syntax highlighting, smart code completion, snippets, code refactoring and integrated Git. Users can change the theme, keyboard shortcuts, preferences and install extensions that add additional functionality.Precisely we are going to talk about the extensions you can install for VS. Here is a list of our favorites1- PythonLink: https://github.com/Microsoft/vscode-pythonPython extension for Visual Studio CodeA Visual Studio Code extension with rich support for the Python language (for all actively supported versions of the language: >=3.6), including features such as IntelliSense (Pylance), linting, debugging, code navigation, code formatting, refactoring, variable explorer, test explorer, and more!NOTE: Web support -- e.g., github.dev -- is limited.Installed extensionsThe Python extension will automatically install the Pylance and Jupyter extensions to give you the best experience when working with Python files and Jupyter notebooks. However, Pylance is an optional dependency, which means that the Python extension will remain fully functional if it is not installed. You can also uninstall it at the expense of some features if you are using a different language server.2- Python IndentLink: https://github.com/kbrose/vsc-python-indentIt is used to correct Python indentation in Visual Studio Code. How it worksEvery time you press the Enter key in a Python context, this extension will parse your Python file down to the location of your cursor, and determine exactly how much to indent the next line (or two in the case of hanging indents) and how much to indent nearby lines.There are three main cases when determining the correct indent. Review the documentation here: https://github.com/kbrose/vsc-python-indent3- Python Doctring GeneratorLink: https://github.com/NilsJPWerner/autoDocstringVisual Studio Code extension to quickly generate docstrings for python functions.FeaturesQuickly generate a docstring fragment that can be tabbed.Choose from several types of docstring formats.Infer parameter types via pep484 type hints, default values and var names.Support for args, kwargs, decorators, errors and parameter types.Docstring FormatsGoogle (default)docBlockrNumpySphinxPEP0257 (coming soon)UsageThe cursor must be on the line directly below the definition to generate a complete auto-populated docstring.Press enter after opening the docstring with triple quotes ("""" or ''')Keyboard shortcut: ctrl+shift+2 or cmd+shift+2 for macCan be changed in Preferences -> Keyboard shortcuts -> extension.generateDocstringCommand: Generate DocstringRight-click menu: Generate DocstringAlso read: 4 Must-Know Python Pandas Functions for Time Series Analysis4- Python ExtendedLink: https://github.com/tushortz/vscode-Python-ExtendedPython Extended is a vscode snippet that makes it easy to write Python code by providing completion options along with all arguments.UsageRun vscode and in a python file, type the name of the method to complete and press tab or enter on selection.How to installOpen vscode. Press F1, search for "ext install" followed by the extension name, in this case "ext install Python Extended" without the ">". Or if you prefer ">ext install", press enter, search for "Python Extended".5- Python PreviewLink: https://github.com/dongli0x00/python-previewA Visual Studio Code extension with debug preview support for the Python language.RequirementsInstall a version of Python 3.6 or Python 2.7. Make sure that the location of your Python interpreter is included in your PATH environment variable.It is best to install the Python extension for Python Intellisense.6- AREPL for PythonLink: https://github.com/almenon/arepl-vscodeAREPL automatically evaluates Python code in real time as you type.UsageFirst, make sure you have python 3.7 or higher installed.Open a python file and click on the cat in the top right bar to open AREPL. You can click the cat again to close it.Or run AREPL via the search command: control-shift-por use the shortcuts: control-shift-a (current document) / control-shift-q (new document)FeaturesReal-time evaluation: no need to run - AREPL evaluates your code automatically. You can control this (or even disable it) in the settings.Variable display: The final state of your local variables is displayed in a collapsible JSON format.Error display: The moment you make a mistake an error is displayed with the stack trace.Settings: AREPL offers many settings to suit your user experience. Customize the look and feel, bounce time, python options and much more.Aldo read: 3 Python Tricks That Will Improve Your Code7- Python PathLink: https://github.com/mgesbert/vscode-python-pathThis extension adds a set of tools to help generate internal import statements in a Python project.Features"Copy Python Path" is accessible from:Command lineExplorer context menuEditor context menuEditor title context menu8- Python Test ExplorerLink: https://github.com/kondratyev-nv/vscode-python-test-adapterThis extension allows you to run your Python Unittest, Pytest or Testplan tests with the Test Explorer user interface.How to get startedInstall the extensionConfigure Visual Studio Code to discover your tests (see the Configuration section and the documentation for the test framework of your choice:Unittest documentationPytest documentationTestplan documentationOpen the sidebar of the test viewExecute your tests via the Run icon in the Test ExplorerFeaturesDisplays a Test Explorer in the test view in the VS Code sidebar with all detected tests and suites and their statusConvenient error reporting during test detectionUnittest, Pytest and Testplan debuggingDisplays the log of a failed test when the test is selected in the explorerTest rerun when saving testsSupports multi-root workspacesSupports Unittest, Pytest and Testplan test frameworks and their plugins9- Python SnippetsLink: https://github.com/ylcnfrht/vscode-python-snippet-packA snippet package to make working with Python more productive This snippet package contains all of the following Python methodsall built-in python snippets and contains at least one example for each methodall python string snippets contain at least one example for each methodall python list snippets contain at least one example for each methodall Python set snippets contain at least one example for each methodall Python tuple snippets contain at least one example for each methodall python dictionary snippets contain at least one example for each methodAnd it contains many other code snippets (such as if/else, for, while, while/else, try/catch, file process, andclass snippets and class examples for oop (polymorphism, encapsulation, inheritance, etc.).If you don't use a method don't worry this extension contains a lot of code examples for each python method.This extension is not just a code snippet, it will also be useful for learning the python programming language.You will learn all python methods with a lot of code examples.For example, if you want to use the string replacement method, you just need to use .replace.But if you don't know how to use the replace method then use string.replace =>10- JupyterLink: https://github.com/Microsoft/vscode-jupyterA Visual Studio Code extension that provides basic notebook support for language kernels that are compatible with Jupyter Notebooks today. Many language kernels will work without any modifications. To enable advanced features, modifications to the VS Code language extensions may be necessary.Notebook supportThe Jupyter Extension uses VS code's built-in notebook support. This interface offers a number of advantages to notebook users:Out-of-the-box support for VS Code's wide range of basic code editing functions, such as hot output, search and replace, and code folding.Editor extensions such as VIM, bracket coloring, linters and many more are available while editing a cell.Deep integration with the general workbench and file-based features of VS Code, such as outline view (table of contents), breadcrumbs, and other operations.Fast load times for Jupyter notebook (.ipynb) files. Any notebook file is loaded and rendered as quickly as possible, while execution-related operations are initialized behind the scenes.Includes a notebook diff tool, which makes it easy to compare and visualize differences between code cells, results and metadata.Extensibility beyond what the Jupyter extension provides. Extensions can now add their own specific language or runtime to notebooks, such as the .NET and Gather interactive notebooks.Although the Jupyter extension comes with a comprehensive set of the most commonly used renderers for output, the marketplace supports installable custom renderers to make working with your notebooks even more productive. To get started writing your own, check out the VS Code renderer api documentation.You can also read data science posts in Spanish here.ConclusionThere are many extensions that you can use with your Visual Studio Code, and deciding which one to use will involve testing, reviewing utilities, use cases and so on in order to make your work easier while coding!Also read: Why Decorators In Python Are Pure Genius?

Join our private community in Discord

Keep up to date by participating in our global community of data scientists and AI enthusiasts. We discuss the latest developments in data science competitions, new techniques for solving complex challenges, AI and machine learning models, and much more!

Join our Discord