IPython

beyond plain python

When executing code in IPython, all valid Python syntax works as-is, but IPython provides a number of features designed to make the interactive experience more fluid and efficient.

Additional resources

This material is adapted from the ICESat2 Hackweek intro-jupyter-git session by @fperez.

First things first: running code, getting help

In the notebook, to run a cell of code, hit Shift-Enter. This executes the cell and puts the cursor in the next cell below, or makes a new one if you are at the end. Alternately, you can use:

  • Alt-Enter to force the creation of a new cell unconditionally (useful when inserting new content in the middle of an existing notebook).

  • Control-Enter executes the cell and keeps the cursor in the same cell, useful for quick experimentation of snippets that you don’t need to keep permanently.

print("Hi")
Hi

Getting help. Use the ? to call up documentation

?
import numpy as np
np.linspace?

To bring up the source code, use ??

np.isclose??

You can also use the * as a wildcard if you know part of the name but not all of it.

*int*?

And if you ever need a summary of the above, you can always use the %quickref magic

%quickref

Tab completion

Tab completion, especially for attributes, is a convenient way to explore the structure of any object you’re dealing with. Simply type object_name.<TAB> to view the object’s attributes. Besides Python objects and keywords, tab completion also works on file and directory names.

# np.

The interactive workflow: input, output, history

2+10
12

The _ stores the value of the last output (which is quite handy if you ran an expensive computation and forgot to assign the output to a variable!)

_+10
22

You can suppress the storage and rendering of output if you append ; to the last cell (this comes in handy when plotting with matplotlib, for example):

10+20;
_
22

The output is stored in _N and Out[N] variables:

_11 == Out[11]
True

Previous inputs are available, too. Either through In[N] or _i for the previous input:

In[11]
'_'
_i
'In[11]'

The history magic prints the input history

%history -n 1-5
   1: print("Hi")
   2: ?
   3:
import numpy as np
np.linspace?
   4: np.isclose??
   5: *int*?

Exercise

Use %history? to have a look at %history’s magic documentation, and write the last 10 lines of history to a file named log.py.

Accessing the underlying operating system

The ! lets you run commands on the terminal

!pwd
/home/runner/work/jupyter-earth/jupyter-earth/docs/jupyter-resources/ecosystem

It doesn’t need to be at the beginning of the cell, and can actually be assigned to a variable.

files = !ls 
print("files this directory:")
print(files)
files this directory:
['binder.md', 'index.md', 'ipython-customization.ipynb', 'ipython.ipynb', 'jupyterbook.ipynb', 'jupyterhub.md', 'jupyterlab.ipynb', 'managing-state.ipynb', 'mauna_loa_data.npz', 'notebook.ipynb', 'widgets.ipynb']
files
['binder.md',
 'index.md',
 'ipython-customization.ipynb',
 'ipython.ipynb',
 'jupyterbook.ipynb',
 'jupyterhub.md',
 'jupyterlab.ipynb',
 'managing-state.ipynb',
 'mauna_loa_data.npz',
 'notebook.ipynb',
 'widgets.ipynb']

Python variables can be passed to the command line by putting them in curly brackets {var}

!echo {files[0].upper()}
BINDER.MD

It can even be in a loop, and with formatting

import os
for i,f in enumerate(files):
    if f.endswith('ipynb'):
        !echo {"%02d" % i} - "{os.path.splitext(f)[0]}"
    else:
        print('--')
--
--
02 - ipython-customization
03 - ipython
04 - jupyterbook
--
06 - jupyterlab
07 - managing-state
--
09 - notebook
10 - widgets

Beyond Python: magic functions

The IPyhton ‘magic’ functions are a set of commands, invoked by prepending one or two % signs to their name, that live in a namespace separate from your normal Python variables and provide a more command-like interface. They take flags with -- and arguments without quotes, parentheses or commas. The motivation behind this system is two-fold:

  • To provide an namespace for controlling IPython itself and exposing other system-oriented functionality that is separate from your Python variables and functions. This lets you have a cd command accessible as a magic regardless of whether you have a Python cd variable.

  • To expose a calling mode that requires minimal verbosity and typing while working interactively. Thus the inspiration taken from the classic Unix shell style for commands.

%magic

Line vs cell magics:

Magics can be applied at the single-line level or to entire cells. Line magics are identified with a single % prefix, while cell magics use %% and can only be used as the first line of the cell (since they apply to the entire cell). Some magics, like the convenient %timeit that ships built-in with IPython, can be called in either mode, while others may be line- or cell-only (you can see all magics with %lsmagic).

A comprehensive list with descriptions is available in the IPython documentation

Let’s see this with some %timeit examples:

%timeit list(range(1000))
13.8 µs ± 62.7 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%%timeit
# comment here

list(range(10))
list(range(100))
1.38 µs ± 4.57 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Line magics can be used even inside code blocks:

for i in range(1, 5):
    size = i*100
    print('size:', size, end=' ')
    %timeit list(range(size))
size: 100 
987 ns ± 12 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
size: 200 
1.67 µs ± 56.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
size: 300 
2.74 µs ± 95.7 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
size: 400 
4.37 µs ± 177 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Magics can do anything they want with their input, so it doesn’t have to be valid Python (note that the below may not work on a Windows machine, depending on how you are running Jupyter on it):

%%bash
echo "My shell is:" $SHELL
echo "My disk usage is:"
df -h
My shell is: /bin/bash
My disk usage is:
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        84G   49G   35G  59% /
devtmpfs        3.4G     0  3.4G   0% /dev
tmpfs           3.4G   12K  3.4G   1% /dev/shm
tmpfs           696M  1.1M  695M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.4G     0  3.4G   0% /sys/fs/cgroup
/dev/loop0       56M   56M     0 100% /snap/core18/2074
/dev/loop1       56M   56M     0 100% /snap/core18/2128
/dev/loop2       71M   71M     0 100% /snap/lxd/21029
/dev/loop3       33M   33M     0 100% /snap/snapd/12398
/dev/loop4       33M   33M     0 100% /snap/snapd/12704
/dev/sdb15      105M  5.2M  100M   5% /boot/efi
/dev/sda1        14G  4.1G  9.0G  32% /mnt
%%writefile test.txt
This is a test file!
It can contain anything I want...

And more...

Writing test.txt
!cat test.txt
This is a test file!
It can contain anything I want...

And more...
%lsmagic
Available line magics:
%alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %conda  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.
def to_optimize(N):
    total = [0,0]
    ta = 0
    tb = 0
    for i in range(N):
        for j in range(N):
            a = i**2
            b = j*2
            total[0] +=  a
            total[1] +=  b
    return total
%timeit to_optimize(1_000)
458 ms ± 3.33 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The prun magic runs a python code profiler

%prun to_optimize(1_000)
 

Running normal Python code: execution and errors

Not only can you input normal Python code, you can even paste straight from a Python or IPython shell session:

>>> # Fibonacci series:
... # the sum of two elements defines the next
... a, b = 0, 1
>>> while b < 10:
...     print(b)
...     a, b = b, a+b
1
1
2
3
5
8
In [1]: for i in range(10):
   ...:     print(i, end=' ')
   ...:     
0 1 2 3 4 5 6 7 8 9 

And when your code produces errors, you can control how they are displayed with the %xmode magic:

%%writefile mod.py

def f(x):
    return 1.0/(x-1)

def g(y):
    return f(y+1)
Writing mod.py

Now let’s call the function g with an argument that would produce an error:

import mod
# mod.g(0)

Basic debugging

When running code interactively, it can be tricky to figure out how to debug…

The debug magic lets you move up (u) or down (d) the stack trace, and you can print out variables to see if you can figure out what went wrong.

%debug
ERROR:root:No traceback has been produced, nothing to debug.
# enjoy = input('Are you enjoying this tutorial? ')
# print('enjoy is:', enjoy)

Running code in other languages with special %% magics

%%perl
@months = ("July", "August", "September");
print $months[0];
July

Plotting in the notebook

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(0, 2*np.pi, 300)
y = np.sin(x**2)
plt.plot(x, y)
plt.title("A little chirp")
Text(0.5, 1.0, 'A little chirp')
../../_images/ipython_70_1.png