HPC Scalability Profiling

Understanding Code Scalability

Scalability Profiling

This session introduces the concept of scalability and how to profile the scalability of your code with an increasing number of cores.


Introduction to Software Engineering (Schmidt)

This workshop will provide an introduction to [PyBaMM](https://pybamm.org/) (Python Battery Mathematical Modelling), an open-source battery simulation package written in Python. It will cover the basic functionality to solve and customise battery models.

# Preparation for PyBaMM workshop

You will be working through the practical exercises using your own laptops. You will need to setup your laptop with a few things, which you may already have. These are:

## Command Line Tool

You will need a command line tool (shell/console) in order to install PyBaMM and run Python scripts. You should have a command line tool already installed on your laptop:
 
On Windows, you can use cmd, Powershell or Git Bash (which is included in Git For Windows package - https://gitforwindows.org/).
On macOS and Linux, you will already have a command line tool available on your system. You can use a command line tool such as Bash (https://www.gnu.org/software/bash/), or any other command line tool that has similar syntax to Bash (e.g. Zsh). 
 
## Python Distribution
 
The material has been developed using the standard Python distribution version 3.8 – 3.10 and is using venv for virtual environments and pip for package management. The material has not been extensively tested with other Python distributions and package managers, but most sections are expected to work with some modifications. For example, package installation and virtual environments would need to be managed differently, but Python script invocations should remain the same regardless of the Python distribution used.
 
To download a Python distribution for your operating system, please head to Python.org (https://www.python.org/downloads/) .
 
## venv and pip
 
If you are using a Python 3 distribution from Python.org, venv and pip will be automatically installed for you. If not, please make sure you have these two tools (that correspond to your Python distribution) installed on your machine.
 
## VSCode or other IDE
 
The course does not assume you are using any particular IDE, and you could use any text editor to edit your Python scripts and run them via the command line. However, we do recommend the use of an IDE when using PyBaMM. For example, you can use Microsoft’s free Visual Studio Code (VSCode) tool.
 
You can download and install VSCode here (https://code.visualstudio.com/). Make sure you install the Python Extension via the “Extension” button on the LHS sidebar (search for “python” in the search bar and install the extension titled “Python” by Microsoft)
 
## PyBaMM

Once you have the above, you can create a virtual environment by typing the following into your command-line.

```bash
python -m venv env
```

This will create a new virtual environment in the subdirectory env. Note that `python` is the name of the python interpreter that you have installed. In some systems you might need to use `python3` instead. If you are using **Windows**, you can then activate your virtual environment by using the following command:

```bash
./env/Scripts/activate
```

If you are using Powershell you might need to capitalise `Activate`.

For **Mac or Linux** systems you can use

```bash
source env/bin/activate
```

Once you have activated your virtual environment, you can install PyBaMM into it using the Python package manager `pip`

```bash
pip install pybamm
```

For more installation information on PyBaMM, or for more options, please consult the PyBaMM documentation here: ​​https://docs.pybamm.org/en/latest/source/user_guide/installation/index.html

You can test your PyBaMM installation by creating a new file `test.py` with the following contents:

```python
import pybamm
import matplotlib.pyplot as plt
model = pybamm.lithium_ion.DFN()
sim = pybamm.Simulation(model)
sim.solve([0, 3600])
fig = sim.plot()
plt.savefig("test.png")
```

Run this script using 

```bash
python test.py
```

And it should generate an image file `test.png`, which contains the result of simulating a DFN model at time $t = 0$



Faraday PyBaMM Training Workshop

An introduction to PyBaMM (Python Battery Mathematical Modelling)

---

## Timetable

The course takes place over five Mondays: 13 May, 20 May, 03 June, 10 June, 17 June.

| Time            | Format            | Location       |
|-----------------|-------------------|----------------|
| 08:00-09:30 UTC+1 | Online (Zoom)   | Singapore      |
| 09:30-10:30 UTC+1 | In person | Oxford         |
| 14:00-15:00 UTC+1 | In person | Oxford         |
| 16:00-17:30 UTC+1 | Online (Zoom)   | North America  |

Note that all times are UTC+1 (British Summer Time):
 - 08:00-09:30 is 15:00-16:30 UTC+8 in Singapore
 - 16:00-17:30 is 11:00-12:30 UTC-4 (EDT) or 08:00-09:30 UTC-7 (PDT) in North America

The in-person sessions will be at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/) except on the 17th June which will be in the [Department of Statistics](https://www.stats.ox.ac.uk/).

---


## Preparation for the course

You will be working through the practical exercises using your own laptops. You will need to set up your laptop with a few things, which you may already have. This course requires that you have installed:

- Python 3
- The `venv` module for Python 3
- The `git` version control package
- A `bash` command-line shell
- VSCode Interactive Development Environment (IDE)
- Docker

If you already have these then great, otherwise please read on for instructions on how to install them on your machine. The instructions below are different depending on the operating system you have installed on your laptop.

### Microsoft Windows

All the material assumes you have a unix-like environment, and the easiest way to get this on a Windows laptop is to install Windows Subsystem for Linux (WSL).

You can install WSL using the instructions here:

https://learn.microsoft.com/en-us/windows/wsl/install

Once you have an Ubuntu 22.04 distribution installed, open the Ubuntu terminal window and use this command to install all the necessary packages (Python, venv and git):

```bash
sudo apt install -y python3-pip python3-venv python3-dev build-essential git
```

We will be using the VSCode IDE for the course. Make sure to install VSCode using the *Windows* installer here: https://code.visualstudio.com/download

Once VSCode has installed, run it and install the "Python (Microsoft)" and the "WSL (Microsoft)" extensions.

Finally, install Docker. If you are unfamiliar with Docker, the easiest way is probably to install [Docker Desktop](https://www.docker.com/products/docker-desktop/).


### macOS

If you have a Mac then we recommend using homebrew to install the dependencies, you can find installation instructions here: https://brew.sh/. Then you can use brew to install everything you need, like so:

```bash
brew install python3 git
brew install --cask visual-studio-code
```

Once VSCode has installed, run it and install the "Python (Microsoft)" extension.

Finally, install Docker. If you are unfamiliar with Docker, the easiest way is probably to install [Docker Desktop](https://www.docker.com/products/docker-desktop/).


### Ubuntu 22.04

The material has been written using Ubuntu 22.04, but if you are running another Linux distribution then you should be able to follow along with minimal changes. If you wish to install Ubuntu 22.04 then you can from here:

https://ubuntu.com/download/desktop

Once Ubuntu is installed, log in and open a command-line terminal. Install the required software via apt by typing in the following command:

```bash
sudo apt install -y python3-pip python3-venv python3-dev build-essential git
```

Then install VSCode using snap:

```bash
sudo snap install --classic code
```

Once VSCode has installed, run it and install the "Python (Microsoft)" extension.

Finally, install Docker. If you are unfamiliar with Docker, the easiest way is probably to install [Docker Desktop](https://www.docker.com/products/docker-desktop/).


## Prerequisites

You will need to have some familiarity with the bash shell, and Python. If you are new to either of those, we recommend making your way through our introductory material:

- https://train.oxrse.uk/material/HPCu/technology_and_tooling/bash_shell
- https://train.oxrse.uk/material/HPCu/introductory_courses/python


## HPC

One session during the course will familiarise you with using a High Performance Computing (HPC) system.
You should try and create an account on your home institution's HPC resource.

**You should do this early as it may take some time to create an account and successfully log in.**

For instance, if you are a fellow at Oxford, you should create an account on [ARC](https://www.arc.ox.ac.uk/home) using the project `dtce-schmidt`. Fellows at other universities should consult their local HPC service.
Please get in touch with us if you cannot create an account on your local HPC service.


Schmidt Sciences Software Engineering

This course provides an introduction to software engineering for 1st year DTC students. It covers the main software architecture paradigms: procedural, object-orientated and functional programming, version control with Git, testing and continuous integration, project packaging and containerisation, an intro to using HPC clusters and computational workflows with snakemake.

It is assumed that students have previous programming experience. The course is taught in Python, but can be taken by students coming from a different language as an introduction to Python is built into the material.

### Week 1 (9/10/2023 - 13/10/2023)
  - Monday: Procedural and Object-orientated programming
  - Tuesday: Functional programming
  - Wednesday: Version control with Git and Github
  - Thursday: Software testing and continuous integration
  - Friday: Python packaging and containerisation with docker

### Week 2 (16/10/2023 - 20/10/2023)
  - Monday: Introduction to HPC clusters
  - Tuesday: Workflows with snakemake
  - Wednesday - Friday: Group projects

### Week 3 (23/10/2023 - 27/10/2023)
  - Monday - Friday: SABS students will start year-long software engineering projects, NERC/DTP students will continue group projects

DTC Software Engineering Course

An introduction to software engineering for 1st year DTC students

## Why Maths?

Maths is the language we use to quantitatively describe the world.

You are going into research, and you

- **may** need to be proficient in maths as a tool for describing what you work on
- **will** have to read and understand papers that use maths

Some of you **may** feel you don't need to know any maths, but this course will be useful to you even if you never have to write down a system of differential equations yourself.

**Data analysis:**

- Interpretation and inference
- Identify patterns, trends, relationships
- Deal robustly with uncertainty and variation

**Describe the behaviour of systems:**

- Remove ambiguity: explicit assumptions
- Quantitative hypotheses
- Make predictions, through simulation and analysis
- "If I make this intervention, I expect to see that change"
- Explain **why** something is observed


**Vital for dynamic and nonlinear systems**

- Simple intuition breaks down
- Most of biology is dynamic and nonlinear!


## Course aims

- Develop confidence in your mathematical abilities
    - Extensive practice
- Become able to communicate effectively with mathematical collaborators
- Ensure you can read and understand mathematical papers in your field
- Build on your ability to apply computational tools from **Python** to solve problems

## Topics covered

- Graphs, and basic tools such as logs
- Calculus: differentiation & integration
- Complex numbers
- Ordinary differential equations
- Linear algebra (matrices)
- Coupled systems of ordinary differential equations
- Use of Python

DTC Essential Maths Course 2023

An introduction to essential mathematics, for 1st year DTC students

slides are [here](https://docs.google.com/presentation/d/1valsEPTBRXl1-QiQRkMnNdwStJ48q7qE36HtvcL6Vk0/edit?usp=sharing)

Version Control with Git

This couse gives an introduction to Git, a popular version control systems used extensivly in academia and industry. 

# Preparation for the course

You will be working through the practical exercises using your own laptops. You will need to setup your laptop with a few things, which you may already have. This course requires that you have installed:

- Python 3
- The `venv` module for Python 3
- The `git` version control package
- A `bash` command-line shell
- VSCode Interactive Development Environment (IDE)

If you already have these then great, otherwise please read on for instructions on how to install them on your machine. The instructions below are different depending on the OS you have installed on your laptop.

## MS Windows

All the material assumes you have a unix-like environment, and the easiest way to get this on a Windows laptop is to install Windows Subsystem (WSL) for Linux.

You can install WSL using the instructions here:

https://learn.microsoft.com/en-us/windows/wsl/install

Once you have an Ubuntu 22.04 distribution installed use the ubuntu terminal and use this command to install all the necessary packages (Python, venv and git):

```bash
sudo apt install -y python3-pip python3-venv python3-dev build-essential git
```

We will be using the VSCode IDE for the course. Make sure to install VSCode using the *Windows* installer here: https://code.visualstudio.com/download

Once VSCode has installed, run it and install the "Python (Microsoft)" and the "WSL (Microsoft)" extensions.


## MacOS

If you have a Mac then we recommend using homebrew to install the dependencies, you can find installation instructions here: https://brew.sh/. Then you can use brew to install everything you need, like so:

```bash
brew install python3 git
brew install --cask visual-studio-code
```

Once VSCode has installed, run it and install the "Python (Microsoft)" extension.


## Ubuntu 22.04

The material has been written using Ubuntu 22.04, but if you are running another Linux distribution then you should be able to follow along with minimal changes. If you wish to install Ubuntu 22.04 then you can from here:

https://ubuntu.com/download/desktop

Once Ubuntu is installed, log in and open a command-line terminal. Install the required software via apt by typing in the following command:

```bash
sudo apt install -y python3-pip python3-venv python3-dev build-essential git
```

Then install VSCode using snap:

```bash
sudo snap install --classic code
```

Once VSCode has installed, run it and install the "Python (Microsoft)" extension.


Software Engineering for bioscientists (DTP)

This course attempts to give an introduction to a useful set of algorithms for scientific computing, along with practical skills in implementation of these algorithms in Python using linear algebra and other scientific libraries. The topics covered are

**Week 1**
- *Monday*: Linear algebra direct solvers and matrix decompositions
- *Tuesday*: Sparse matrices and iterative solvers
- *Thursday*: Solution of Ordinary Differential Equations
- *Friday*: Non-linear Optimisation

**Week 2**
- *Monday*: Bayesian Inference and MCMC sampling
- *Tuesday - Friday*: Individual projects

On the software side, we will focus exclusively on implementation in Python, using the Numpy library for multidimensional arrays, and the Scipy library for both sparse matrices and many, many other general purpose scientific computing algorithms.

Scientific Computing

This course gives an introduction to a useful set of algorithms for scientific computing, including: Linear algebra direct solvers and matrix decompositions; Sparse matrices and iterative solvers; Solution of Ordinary Differential Equations; Non-linear Optimisation; Bayesian Inference and MCMC sampling

PyBaMM Developer Training

test

Procedural Programming

Variables

Functions

Containers

Arrays

Procedural Programming is based around the idea that code should be structured
into a sequence of **procedures** that operate on data. This course will
introduce you to the basics of procedural programming in either Python or C++.


Object-Orientated Programming

Classes

Inheritance and Composition

Polymorphism

The Object Oriented Paradigm builds upon the Procedural Paradigm, but builds code around data.
This course will introduce you to the basics of Object Oriented Programming in either Python or C++.


Functional Programming

Side Effects Cpp

Side Effects

Recursion

Higher Order Functions

Functional Programming is based around the idea that programs are constructed
by applying and composing/chaining **functions**. This course will introduce
you to the basics of functional programming in either Python or C++.


Software Architecture and Design

Programming design concepts and patterns, such as procedural, object-orientated, functional, generic. Covers data structures, memory allocation, ownership.


The Bash shell

Introducing the Shell

Files and Directories

Creating Things

Pipes and Filters

Shell Scripts

Loops

The Bash shell is a command-line interface used in Unix-based operating systems
such as Linux and macOS. This course will introduce you to the basics of using
the Bash shell.


Version Control

Automated Version Control

Setting Up Git

Creating a Repository

Tracking Changes

Exploring History

Remote Repositories

Branches

Ignoring Things

This course introduces the basics of version control using the Git version control system.
We will learn how to setup Git, and use it to track changes in our code.


Packaging and Dependency Management

Packaging Python Projects

Making Packages

Reusing Packages

Sharing Packages

Virtual Environments

Introduction to CMake

This course introduces the basics of packaging and dependency management in Python and C++.
For Python, we introduce `venv` for virtual envronments, and `pip` for package management and how to structure a modern Python package and publish it to PyPI.
For C++, we introduce the CMake build system and how use it to manage dependencies and the build process.


Best Practices

Code Style

Linters

This course covers how to style your Python code, and use linters to enforce a consistant style and highlight any code that can lead to commonly encountered bugs or problems.


IDEs

VSCode

Integrated Development Environments (IDEs)provide programmers with a complete development environment to
write, edit, debug, and deploy their code. This course introduces the popular VSCode IDE, both for Python
and C++ development.


Testing

Automated Testing

Diagnosing Issues

Scaling Up Unit Testing

This course introduces the basics of automated testing and debugging in Python, using the Pytest framework.


Containerisation with Docker

Introduction

Docker Command Line

Cleaning Up Containers

Running-containers

Finding Containers on Docker Hub

Creating Containers

More Complex Containers

Examples

Using Docker with Github Actions

Using Docker with Jekyll

Reproducibility and Granularity

Setup

This course aims to introduce the use of Docker containers with the goal of
using them to effect reproducible computational environments.


Snakemake Tutorial

Basics: An example workflow

Advanced: Decorating

Additional features

Short tutorial

This tutorial introduces the text-based workflow system
Snakemake.


Technology and Tooling

Programming design concepts and patterns, such as proceedural, object-orientated, functional, generic. Covers data structures, memory allocation, ownership.


Collaboration on GitHub

Git Refresher

Collaborative Worklow

Issue Management

This course will show you how to use Git for effective collaboration with others
using the GitHub platform. After completion of the course, you will be able to
contribute to projects on GitHub, including opening issues and pull requests.


Continuous Integration

GitHub Actions

Code Coverage

Documentation

This course introduces the concept of continuous integration and how to set it up for a Python project using GitHub Actions.


Software Project Management

Courses on software project management such as on collaborative programming and continuous integration.


Introduction to High Performance Computing

Why Use a Cluster?

Connecting to Remote HPC systems

Exploring Remote Resources

Introduction to Job Scheduling

Accessing Software via Modules

Transferring Files with Remote Computers

Running a parallel job

Using Resources Effectively

Using Shared Resources Responsibly

An introduction to high-performance computing (HPC), covering connecting to HPC resources and the slurm job scheduler.


Introduction to Parallelism

This session introduces the basic concept of parallelism, and the two main paradigms - shared and distributed memory.


Introduction to the Message Passing Interface

Communicating Data in MPI

Point-to-Point Communication

Collective Communication

Non-blocking Communication

Advanced Communication Techniques

A Common Communication Patterns

Porting Serial Code to MPI

Optimising MPI Applications

This session introduces the Message Passing Interface, and shows how to use it to parallelise your code.


High Performance Computing

An introduction to high-performance computing (HPC), covering connecting to HPC resources, job schedulers (also known as batch systems), and techniques for determining the scalability of code on HPC resources.


Intro to Python

Running Python

Variables and Types

Writing and running Python from an IDE

Built-in Functions and Help

Libraries

Analyzing and Visualizing Data

Pandas Dataframes

Lists

For Loops

Conditionals

Looping Over Data Sets

Errors and Exceptions

Writing Functions

Variable Scope

Project: Snake Game

This course introduces the basics of programming in Python.
We will learn how to run Python code, and how to use variables, functions, and control flow.
We will also learn how to use Python for data analysis and visualization.


Introductory Courses

Introductory courses on programming languages.


Essential Maths

Graphs

Indices, logs and exponentials

Differentiation 1

Differentiation 2

Differentiation 3

Integration 1

Integration 2

Complex numbers

Differential equations 1

Differential equations 2

Differential equations 3

Linear algebra 1

Linear algebra 2

Systems of differential equations 1

Systems of differential equations 2

Systems of differential equations 3

Probability 1

Probability 2

Probability 3

A course in maths, containing a primer in everything you need to know in order to analyse systems of differential equations.


Linear Algebra Solvers

Matrix form of equations

Gaussian Elimination

LU decomposition

Cholesky decomposition

QR decomposition

This course covers the basics of linear algebra focusing on the solution of
dense systems of linear equations using direct methods and matrix
decompositions. Both the theory and implementation of these methods are
covered, as well as some standard Python libraries for linear algebra.


Sparse Matrices and Iterative Solvers

Sparse Matrices

COOrdinate format

Finite Difference Matrix

Scipy.sparse and problems

Jacobi and Relaxation Methods

Krylov subspace methods and CG

This course covers the basics of iterative methods for
solving linear systems, and how to use sparse matrices to represent and solve
these systems efficiently.


Non-linear Optimisation

Line Search Methods

Trust Region Methods

Derivative-free methods

Courses on scientific computing on subjects including linear algebra, ode solving, bayesian statistics etc.