In: computer, programming, language.

Python (programming language)

Appeared: February 1991
Typing: duck, dynamic, strong typing; gradual
Implementations: CPython, PyPy, Stackless Python, MicroPython, CircuitPython, IronPython, Jython
Dialects: Cython, RPython, Starlark

Python is an interpreted high-level general-purpose programming language. Its design philosophy emphasizes code readability with its notable use of significant indentation.

Personal

I discovered Python in 2007 and fell in love instantly. I was working on a small game and Python made it really easy.
A year later I had a job where I used my Python skills a lot and became proficient. I used Python everywhere since then.
I did all kinds of projects with Python, anything you can imagine really, it’s a versatile language.

For X years in a row, Python was the most wanted language on Stackoverflow, which has to mean something:

Python is no #1 in Tiobe index since 2021:
https://tiobe.com/tiobe-index

Versions

CPython

The default implementation

PyPy

Built using the RPython language
The main reason to use it instead of CPython is speed: it generally runs faster
https://pypy.org

Stackless

Stackless Python programming language
https://github.com/stackless-dev/stackless

MicroPython

Optimised to run on microcontrollers and in constrained environments
Includes a small subset of the Python standard library
https://microPython.org
https://github.com/micropython/micropython
https://github.com/mcauser/awesome-micropython

Pyodide

Pyodide is a Python distribution for the browser and Node.js based on WebAssembly
https://pyodide.org
https://github.com/pyodide/pyodide
https://blog.pyodide.org

RustPython

Python Interpreter written in Rust
Compiled to WASM, runs in the browser
https://rustpython.github.io
https://github.com/RustPython/RustPython

Brython

Python 3 implementation for client-side web programming
https://brython.info
https://pypi.org/project/brython

Build & distribute

Overview of Python Packaging !
https://packaging.python.org/en/latest/overview

zipapp - Executable Python ZIP archives
https://docs.python.org/3/library/zipapp.html

.pyz and .pyzw extensions: “Python ZIP Applications” and “Windowed Python ZIP Applications”
https://peps.python.org/pep-0441

Fully self-contained Python zipapps as outlined in PEP 441
https://github.com/linkedin/shiv

.pex (Python EXecutable) files, executable Python environments in the spirit of virtualenvs
https://github.com/pex-tool/pex

PyInstaller bundles a Python application and all its dependencies into a single package.
https://pyinstaller.org

cx_Freeze creates standalone executables from Python scripts, with the same performance, is cross-platform, and should work on any platform that Python itself works on.
https://marcelotduarte.github.io/cx_Freeze

Nuitka is a Python compiler written in Python. You feed it your Python app, it does a lot of clever things, and spits out an executable or extension module.
https://github.com/Nuitka/Nuitka

PyOxidizer is capable of producing a single file executable - with a copy of Python and all its dependencies statically linked and all resources (like .pyc files) embedded in the executable
https://github.com/indygreg/PyOxidizer

Tool for creating installers from Conda packages.
https://github.com/conda/constructor
https://conda.github.io/constructor

Kivy: The Open Source Python App Development Framework.
Build and distribute beautiful Python cross-platform GUI apps with ease.
https://kivy.org

BeeWare. Write once, deploy everywhere.
Write your apps in Python and release them on iOS, Android, Windows, MacOS, Linux, Web, and tvOS using rich, native user interfaces.
https://beeware.org

pyproject.toml

This was always a pain in the ass for me… Keeping the documentation close.

https://peps.python.org/pep-0621
https://packaging.python.org/en/latest/guides/writing-pyproject-toml
https://packaging.python.org/en/latest/specifications/pyproject-toml

setup.py

https://packaging.python.org/en/latest/guides/modernize-setup-py-project
https://packaging.python.org/en/latest/discussions/setup-py-deprecated

Libraries

Debugging and Profiling

Articles about Python memory profiling 📖 📚 ::

Checking

Ruff

Extremely fast Python linter, written in Rust
Ruff can be used to replace Flake8 (plus a variety of plugins)
https://github.com/charliermarsh/ruff

Austin

Frame stack sampler for CPython written in pure C
Samples are collected by reading the CPython interpreter virtual memory space in order to retrieve information about the currently running threads along with the stack of the frames that are being executed
Make powerful statistical profilers that have minimal impact on the target application and that don’t require any instrumentation
https://github.com/p403n1x87/austin

Guppy

Heap analysis toolset

line-profiler & kernprof

Line-by-line profiling of functions
kernprof is a convenient script for running either lineprofiler or the Python standard library’s cProfile or profile modules, depending on what is available

Memory profiler

Monitor memory consumption of a process as well as line-by-line analysis of memory consumption for python programs
Pure Python module which depends on the psutil module

Memray

Track memory allocations in Python code, in native extension modules, and in the Python interpreter itself
Generate several different types of reports to help you analyze the captured memory usage data
https://github.com/bloomberg/memray

Pympler

Measure, monitor and analyze the memory behavior of Python objects in a running Python application
Written entirely in Python, with no ext dependencies

Pyroscope

Continuous Profiling Platform. Debug performance issues down to a single line of code.

pyroscope exec python manage.py runserver # If using Python

py-spy

Sampling profiler for Python programs
Visualize what your Python program is spending time on without restarting the program or modifying the code in any way
Is extremely low overhead: it is written in Rust for speed and doesn’t run in the same process as the profiled Python program.

Samply

Command-line sampling profiler, which uses the Firefox profiler UI.
Written in Rust.
https://github.com/mstange/samply

Scalene

High-performance, high-precision CPU, GPU, and memory profiler for Python
Scalene has both a CLI and a web-based GUI
It runs" orders of magnitude faster" than many other profilers while delivering far more detailed information
It is also the first profiler ever to incorporate AI-powered proposed optimizations
https://github.com/plasma-umass/scalene

Snoop

A powerful set of Python debugging tools, based on PySnooper
Just import snoop and @snoop the function you want to snoop
https://github.com/alexmojaki/snoop

Scraping

AutoScraper

Smart, automatic web scraper
https://github.com/alirezamika/autoscraper

CoCrawler

Versatile web crawler built using modern tools and concurrency
https://github.com/cocrawler/cocrawler

Grab

Web scraping framework

HtmlMatch

Automatic data scraping
https://github.com/santinic/htmlmatch

Pattern

Web mining (Google, Twitter and Wikipedia API, a web crawler, a HTML DOM parser), natural language processing (part-of-speech taggers, n-gram search, sentiment analysis, WordNet), machine learning (vector space model, clustering, SVM), network analysis and Canvas visualization

PySpider

A Spider (Web Crawler) System in Python
https://github.com/binux/pyspider

Ruia

Async Python 3 web scraping micro-framework based on asyncio

Scrapy

Framework for extracting the data you need from websites

Image

EXIF read/ write:

Articles

Python has too many package managers

By Larry Du, posted July 6 2024
https://dublog.net/blog/so-many-python-package-managers

Thoughts on the Python packaging ecosystem

By Pradyun Gedam, posted January 21 2023
https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging

The origins of Python

By Lambert Meertens, posted November 2022
https://inference-review.com/article/the-origins-of-python

Python behind the scenes: How variables are implemented in CPython

https://tenthousandmeters.com/blog/python-behind-the-scenes-5-how-variables-are-implemented-in-cpython

Python behind the scenes: How the Python import system works

https://tenthousandmeters.com/blog/python-behind-the-scenes-11-how-the-python-import-system-works

Writing Python extensions in Assembly

https://tonybaloney.github.io/posts/extending-python-with-assembly.html

Asynchronous web scraping: Scaling for the moon

https://scrapecrow.com/asynchronous-web-scraping.html
Take a look at asynchronous python for scraping
Sync, async, async with throttler, async limiter with max rate

When your data doesn’t fit in memory: the basic techniques

Technique #1: Compression
technique #2: Chunking, loading all the data one chunk at a time
technique #3: Indexing, when you need a subset of the data
https://pythonspeed.com/articles/data-doesnt-fit-in-memory/

About memory in Python

Python type checkers to keep your code clean

Mypy, Pytype, Pyright/ Pylance, and Pyre/ Pysa can help you keep your type-hinted Python code bug-free
https://infoworld.com/article/3575079/4-python-type-checkers-to-keep-your-code-clean.html

Learn

Scientific Computing with Python, 300 hours, certification:
https://freecodecamp.org/learn/scientific-computing-with-python/

Machine Learning with TensorFlow, 300 hours, certification:
https://freecodecamp.org/learn/machine-learning-with-python/

Awesome

Refs

×