Python

Table of Contents

1 TODO

1.1 TODO define a static method of a class? so that I can avoid calling of self

1.2 TODO define an inline function to avoid passing constant parameter: the scope of variable?

1.3 TODO python debugger for emacs

1.4 TODO a tree reader for leetcode

2 Language

The ultimate reference:

For python 3

atom is the most basic expression, while the enclosure is interesting. 

atom      ::=  identifier | literal | enclosure
enclosure ::=  parenth_form | list_display | dict_display | set_display
               | generator_expression | yield_atom

The display is a special syntax for python to create lists and dicts, using comprehension. 

list_display ::=  "[" [starred_list | comprehension] "]"
set_display ::=  "{" (starred_list | comprehension) "}"

dict_display       ::=  "{" [key_datum_list | dict_comprehension] "}"
key_datum_list     ::=  key_datum ("," key_datum)* [","]
key_datum          ::=  expression ":" expression | "**" or_expr
dict_comprehension ::=  expression ":" expression comp_for

And the comprehension grammar: 

comprehension ::=  expression comp_for
comp_for      ::=  [ASYNC] "for" target_list "in" or_test [comp_iter]
comp_iter     ::=  comp_for | comp_if
comp_if       ::=  "if" expression_nocond [comp_iter]

The comp_for non-terminal contains one or more for clause, and zero or more if clause.

The evaluation of these displays will evaluate from left to right, so the last assignment will prevail in the case of set and dict. The comprehension is executed by nesting the for and if clauses from left to right.

So some examples: 

((a,b) for a in range(2) for b in range(3))
[(a,b) for a in range(2) for b in range(3)]
{(a,b) for a in range(2) for b in range(3)}

{a:b for a in (1,2) for b in (3,4)}

Note the out-most braces are required, which also creates a scope.

3 Emacs support

Install the elpy package. It provides:

  • C-c C-c runs the shell and send the current buffer
  • C-c C-d runs elpy-doc
  • C-c C-t runs elpy-test, which runs the unittest discover

To enable linter python in emacs, use pylint. It will use pylint executable. And it also needs the configure file. Generate it: 

pylint --generate-rcfile > ~/.pylintrc

4 Unit Test

class MyTest(unittest.TestCase):
    def test_me(self):
        self.assertEqual(1,2)
unittest.main()

python unit test can support automatic test discovery. To use that, the file must be named test_xxx.py, and run the python -m unittest discover.

5 Concept

5.1 Scoping

There're four levels:

  • current scope
  • parent scope
  • module scope (global)
  • built-in scope

nonlocal keyword specify this variable should be referenced to the parent scope. But, this will not reach global. Instead, the global keyword declares the listed variables to be in the module level scope.

The nonlocal statement causes the listed identifiers to refer to previously bound variables in the nearest enclosing scope excluding globals.

As an example: 

var = 0 # global

def outer():
  var = 1 # parent
  def inner():
    nonlocal var
    var = 2 # local
    global var
    var =3
  inner()
  # var = 2

outer()
# global var = 3

5.2 Collection

5.2.1 String

5.2.1.1 Concatenation
  • concatenate two strings directly by +.
  • need to convert integer to string before concatenate: s + str(35)
  • "".join(lst) works
5.2.1.2 split
str.split(sep=None)
default by white space
str.strip()
strip out white space at both begin and end
str.replace(old, new)
replace all.
str.startswith(s)
str.endswith(s)
5.2.1.3 Slicing

String is an immutable object. It can use slicing. E.g. reversing a string is as easy as "hello"[::-1]!

However, notice that when using a negative step, the slicing should be lst[end:begin:-1]. This is because x = i + n*k:

with a third “step” parameter: a[i:j:k] selects all items of a with index x where x = i + n*k, n >= 0 and i <= x < j.

Also, the negative step does not always work as expect. E.g. the i index is included and j is not; the j can not be negative, then how can I include the first one in the list??

Thus if want to get a reverse of a sub-string, I would get sub-string first and then reverse it.

5.2.2 TODO tuple

5.2.3 List

5.2.3.1 Slicing

The slicing syntax is l[start:end:step]. The slicing will return a new list. Change to that list will not change the original one. 

l[4]
l[4:]
l[::2]
l[:-1]

However, assign to the slicing itself will change the original one: 

l[1:2] = [4,5,6]

Also, assign to a new variable only assign the reference: 

a = [1,2,3]
b = a # only a reference
5.2.3.2 create a list
  • range(stop)
  • range(start, stop[, step])

Creating a matrix: 

newmat=[[-1 for x in range(height)] for y in range(width)]
5.2.3.3 Modify a list
  • list.append
  • list.pop

5.2.4 Dictionary

Create: 

x = {'a': 1, 'b': 2}

Dictionary is not sorted. Use collections.OrderedDict if you want this feature. Basically it remember the order when the elements are inserted. 

import collections
od = collections.OrderedDict(sorted(d.items()))

Merge two dictionary (x and y): 

z = x.copy()
z.update(y)

5.2.5 Set

s = set()
s.add(x)
if x in s:
  pass

5.3 Algorithm

5.3.1 TODO sort

sort a dictionary by value: 

sorted(dict1, key=dict1.get) # => list
sorted(dict1, key=dict1.get, reverse=True)

5.4 Function

5.4.1 variadic parameter

use *args syntax, and args will be a tuple: 

  def foo(*args):
    for a in args:
      print a

use **args to capture all keyword arguments. 

def bar(**kwargs):
  for a in kwargs:
    print a, kwargs[a]

Combine them together: 

def foobar(kind, *args, **kwargs):
  pass

Also, there's a concept for the reverse thing: unpack argument list from a list, with *list: 

def foo(a,b):
  pass

l = [1,2]
foo(*l)

on python3, this syntax can appear on left side 

first, *rest = [1,2,3,4]
first,*l,last = [1,2,3,4]

5.5 Exception

To give a quick feel: 

try:
  pass
except TypeError as e: # capture the exception into a variable
  pass
except AnotherError: # does not capture
  pass
except: # all exception
  pass
else: # if doesn't raise an exception
  pass
finally:
  pass

5.6 Lambda

lambda x : x+2
lambda x: x%2==0

The usage of lambda is often in map and filter.

  • map(lambda_exp, mylist) will execute the lambda expression on each element of the list, and return a list containing the results.

5.7 Packaging

Exposing API: the following only expose foo but not bar. 

__all__ = ['foo']
def foo():
  pass
def bar():
  pass

5.7.1 importing

The local structure directory must contain the __init__.py file to be able to import. 

|-- main.py
|-- mypackage
    |-- __init__.py
    |-- a.py
    |-- b.py
    |-- subdir
        |-- __init__.py
        |-- c.py

The import statements should be: 

from mypackage import a
from mypackage.b import foo as myfoo
from mypackage.subdir import c

5.8 Thread

from threading import Thread

class MyThread(Thread):
  def __init__(self, arg):
    Thread.__init__(self)
    self.arg = arg
  def run(self):
    pass

t = MyThread(arg)
t.start()

6 Type

6.1 Boolean

  • not True

6.2 Integer

  • i += 1

6.3 conversion

  • string to integer: int('45')
  • integer to string: str(45)
  • ASCII to char: chr(100) returns 'd'
  • char to ASCII: ord('d') returns 100

7 Black Tech

If else or: 

var = d.get('key') or 0
# is equal to:
var = d.get('key') if d.get('key') else 0

list comprehension 

even_squares = [x**2 for x in l if x%2 == 0]

8 Pep8

Indent:

  • function and class should be separated by 2 lines
  • In a class, function should be separated by 1 line
  • 1 space before and after variable assignment

Naming

  • function, variable, attribute: func_var_attr
  • protected instance attributes: _protected_field
  • private instance attributes: __private_field
  • class and exception: ClassExceptionName
  • module level constants: CONSTANT
  • instance method of class should use self as first parameter, refer to the object
  • class method should use cls as first parameter, refer to the class

Expression

use DONT use
a is not b not a is b
if not list if len(list) == 0

Import

  • always use absolute path
  • if must use relative, use from . import foo instead of import foo

8.1 document

One can use one line or multi-line document. The doc string can be retrieved by func.__doc__. 

def func():
  """one line doc"""

def func():
  """The outline

  The above empty line is required.
  Here's the detailed documentation.
  """

9 IO

print('xxx', end='')

  f = open('text.txt')
  f.read() # return all content

  f = open('text.txt')
  for line in f:
      print(line)

  with open('a.txt') as f:
      for line in f:
          print(line)

read from stdin: 

for line in sys.stdin:
  print(line)

get command line argument: sys.argv

10 Operating System

10.1 Work filesystem:

import os
for root,dirs,files in os.walk('.'):
  for f in files:
    print f
  • os.path.abspath('relative/path/to/file')
  • os.path.exists("/path/to/file")
  • os.rename('old', 'new')
  • os.path.isfile

10.2 Shell command

os.system
simply run command
os.system("some command")
os.popen
access to input output
stream = os.popen("some command")
stream.read()
  • subprocess.Popen
p = subprocess.Popen("echo Hello World", shell=True, stdout=subprocess.PIPE)
p.stdout.read()
s = subprocess.check_output('wc -l', stdin=p.stdout)
subprocess.call
this is the same as subprocess.Popen except that it waits and gives return code.
return_code = subprocess.call("echo Hello World", shell=True, stdout=subprocess.DEVNULL)

11 Standard Library

11.1 math

  • math.pow
  • math.log

12 Third party libraries

12.1 argparse

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-q', '--query', help='query github api', require=True)
parser.add_argument('-d', '--download', help='do download', action='store_true')
args = parser.parse_args()

12.2 urllib

from urllib import request
import json

url = 'https://api.github.com'
api = '/search/repositories'
query = 'language:C&stars:>10&per_page='+size
response = request.urlopen(url+api+"?q="+query)

s = response.read().decode('utf8')
j = json.loads(s)
# j will be a mix of list and dict

12.3 XML

import xml.etree.ElementTree as ET
root = ET.fromstring(s)
# XPath
nodes = root.findall('{http://www.sdml.info/srcML/src}function')
for node in nodes:
  # do with node
  pass

APIs

  • node.find(XPath)
  • node.findall(XPath)
  • node.get(Attribute)
  • node.text