---
title: "Lec 21 - Apache Arrow"
subtitle: "<br/> Statistical Computing and Computation"
author: "Sta 663 | Spring 2022"
date: "<br/> Dr. Colin Rundel"
output:
  xaringan::moon_reader:
    css: ["slides.css"]
    lib_dir: libs
    nature:
      highlightLines: true
      highlightStyle: solarized-light
      countIncrementalSlides: false
      ratio: "16:9"
---
exclude: true

```{python setup}
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import scipy

import os

plt.rcParams['figure.dpi'] = 200

np.set_printoptions(
  edgeitems=30, linewidth=200,
  precision = 5, suppress=True
  #formatter=dict(float=lambda x: "%.5g" % x)
)

pd.set_option("display.width", 130)
pd.set_option("display.max_columns", 10)
pd.set_option("display.precision", 6)
```

```{r r_setup}
knitr::opts_chunk$set(
  fig.align="center",
  cache=FALSE
)

library(lme4)
```

```{r hooks}
local({
  hook_err_old <- knitr::knit_hooks$get("error")  # save the old hook
  knitr::knit_hooks$set(error = function(x, options) {
    # now do whatever you want to do with x, and pass
    # the new x to the old hook
    x = sub("## \n## Detailed traceback:\n.*$", "", x)
    x = sub("Error in py_call_impl\\(.*?\\)\\: ", "", x)
    #x = stringr::str_wrap(x, width = 100)
    hook_err_old(x, options)
  })
  
  hook_warn_old <- knitr::knit_hooks$get("warning")  # save the old hook
  knitr::knit_hooks$set(warning = function(x, options) {
    x = sub("<string>:1: ", "", x)
    #x = stringr::str_wrap(x, width = 100)
    hook_warn_old(x, options)
  })
  
  hook_msg_old <- knitr::knit_hooks$get("output")  # save the old hook
  knitr::knit_hooks$set(output = function(x, options) {
    x = stringr::str_replace(x, "(## ).* ([A-Za-z]+Warning:)", "\\1\\2")
    x = stringr::str_split(x, "\n")[[1]]
    #x = stringr::str_wrap(x, width = 120, exdent = 3)
    x = stringr::str_remove_all(x, "\r")
    x = stringi::stri_wrap(x, width=120, exdent = 3, normalize=FALSE)
    x = paste(x, collapse="\n")
    
    #x = stringr::str_wrap(x, width = 100)
    hook_msg_old(x, options)
  })
})
```

---

## Apache Arrow

> Apache Arrow is a software development platform for building high performance applications that process and transport large data sets. It is designed to both improve the performance of analytical algorithms and the efficiency of moving data from one system or programming language to another.
>
> A critical component of Apache Arrow is its in-memory columnar format, a standardized, language-agnostic specification for representing structured, table-like datasets in-memory. This data format has a rich data type system (included nested and user-defined data types) designed to support the needs of analytic database systems, data frame libraries, and more.

```{r echo=FALSE, out.width='40%', fig.align="center"}
knitr::include_graphics("imgs/arrow_cols.png")
```

---

## Language support

.pull-left-narrow[
Core implementations in:

* C
* C++
* C#
* go
* Java
* JavaScript
* Julia
* Rust
* MATLAB
* Python
* R
* Ruby
]

.pull-right-wide[
```{r echo=FALSE, out.width='80%', fig.align="center"}
knitr::include_graphics("imgs/arrow_memory.png")
```
]

---

## pyarrow

```{python}
import pyarrow as pa
```

--

The basic building blocks of Arrow are `array` and `table` objects, arrays are collections of data of a uniform type.

.pull-left[
```{python}
num  = pa.array([1, 2, 3, 2], type=pa.int8())
num

year = pa.array([2019,2020,2021,2022])
year
```
]

--

.pull-right[
```{python}
name = pa.array(
  ["Alice", "Bob", "Carol", "Dave"],
  type=pa.string()
)
name
```
]

---

## Tables

A table is created by combining multiple arrays together to form the columns while also attaching names for each column.

```{python}
t = pa.table(
  [num, year, name],
  names = ["num", "year", "name"]
)

t
```


.footnote[`table` is part of pyarrow but not part of the arrow standard, more on this in a bit]

---

## Array indexing

Elements of an array can be selected using `[]` with an integer index or a slice, the former returns a typed scalar the latter an array.

.pull-left[
```{python, error=TRUE}
name[0]
name[0:3]
name[:]
```
]

--

.pull-right[
```{python, error=TRUE}
name[-1]
name[::-1]
name[4]
name[0] = "Patty"
```
]

.footnote[Arrow arrays are immutable and as such do allow for element assignment.]

---

## Data Types

The following types are language agnostic for the purpose of portability, however some differ slightly from what is available from Numpy and Pandas (or R),

* **Fixed-length primitive types**: numbers, booleans, date and times, fixed size binary, decimals, and other values that fit into a given number
  * Examples: `bool_()`, `uint64()`, `timestamp()`, `date64()`, and many more

* **Variable-length primitive types**: binary, string

* **Nested types**: list, map, struct, and union

* **Dictionary type**: An encoded categorical type


.footnote[See [here](https://arrow.apache.org/docs/python/api/datatypes.html#api-types) for the full list of types.]

---

## Schemas

A data structure that contains information on the names and types of columns for a table (or record batch),

.pull-left[
```{python}
t.schema
```
]

.pull-right[
```{python}
pa.schema([
  ('num', num.type),
  ('year', year.type),
  ('name', name.type)
])
```
]

---

## Schema metadata

Schemas can also store additional metadata (e.g. codebook like textual descriptions) in the form a string:string dictionary,

```{python}
new_schema = t.schema.with_metadata({
  'num': "Favorite number",
  'year': "Year expected to graduate",
  'name': "First name"
})
new_schema

t.cast(new_schema).schema
```

---

## Missing values / None / NANs

.pull-left[
```{python}
pa.array([1,2,None,3])
pa.array([1.,2.,None,3.])
```
]

.pull-right[
```{python}
pa.array(["alice","bob",None,"dave"])
pa.array([1,2,np.nan,3])
```
]

--

<div>

.pull-left[
```{python}
pa.array([1,2,None,3])[2]
pa.array([1.,2.,None,3.])[2]
```
]

.pull-right[
```{python}
pa.array(["alice","bob",None,"dave"])[2]
pa.array([1,2,np.nan,3])[2]
```
]

</div>

---

## Nest type arrays

.pull-left[
list type:
```{python}
pa.array([[1,2], [3,4], None, [5,None]])
```
]

--

.pull-right[
struct type:
```{python}
pa.array([
  {'x': 1, 'y': True, 'z': "Alice"},
  {'x': 2,            'z': "Bob"  },
  {'x': 3, 'y': False             }
])
```
]

.footnote[See also [map](https://arrow.apache.org/docs/python/data.html#map-arrays) and [union](https://arrow.apache.org/docs/python/data.html#union-arrays) arrays.]

---

## Dictionary array

A dictionary array is the equivalent to a factor in R or pd.Categorical in Pandas,

.pull-left[ .small[
```{python}
levels = pa.array(['sun', 'rain', 'clouds', 'snow'])
values = pa.array([0,0,2,1,3,None])
dict_array = pa.DictionaryArray.from_arrays(values, levels)
dict_array
dict_array.type
```
] ]

--

.pull-right[ .small[
```{python}
dict_array.dictionary_decode()
levels.dictionary_encode()
```
] ]

---

## Record Batches

In between a table and an array Arrow has the concept of a Record Batch - which represents a chunk of the larger table. They are composed of a named collection of equal-length arrays.

```{python}
batch = pa.RecordBatch.from_arrays(
  [num, year, name],
  ["num", "year", "name"]
)

batch
```

.pull-left[
```{python}
batch.num_columns
batch.num_rows
```
]

.pull-right[
```{python}
batch.nbytes
batch.schema
```
]

---

## Batch indexing

`[]` can be used with a Record Batch to select columns (by name or index) or rows (by slice), additionally the `slice()` method can be used to select rows.

.pull-left[
```{python, error=TRUE}
batch[0]
batch["name"]
```
]

--

.pull-right[
```{python}
batch[0:2].to_pandas()
batch.slice(0,2).to_pandas()
```
]

---

## Tables vs Record Batches

As mentioned previously, `table` objects are not part of the Arrow specification - rather they are a convenience tool provided to help with the wrangling of multiple Record Batches.

```{python}
table = pa.Table.from_batches([batch] * 3)
table
```

--

.pull-left[
```{python}
table.num_columns
table.num_rows
```
]

.pull-right[
```{python}
table.to_pandas()
```
]

---

## Chunked Array

The columns of `table` are therefore composed of the columns of each of the batches, these are stored as ChuckedArrays instead of Arrays to reflect this.

.pull-left[
```{python}
table["name"]
```
]

.pull-right[
```{python}
table[1]
```
]

---

## Arrow + NumPy

Conversion between NumPy arrays and Arrow arrays is straight forward,

```{python}
np.linspace(0,1,11)
pa.array( np.linspace(0,1,6) )

pa.array(range(10)).to_numpy()
```

---

## NumPy & data copies

```{python, error=TRUE}
pa.array(["hello", "world"]).to_numpy()
pa.array(["hello", "world"]).to_numpy(zero_copy_only=False)
```

--

&nbsp;

```{python, error=TRUE}
pa.array([1,2,None,4]).to_numpy()
pa.array([1,2,None,4]).to_numpy(zero_copy_only=False)
```

--

&nbsp;

```{python, error=TRUE}
pa.array([[1,2], [3,4], [5,6]]).to_numpy()
pa.array([[1,2], [3,4], [5,6]]).to_numpy(zero_copy_only=False)
```

---

## Arrow + Pandas

We've already seen some basic conversion of Arrow table objects to Pandas, the conversions here are a bit more complex than with NumPy due in large part to how Pandas handles missing data.

.pull-left[ .small[
| Source (Pandas)       | Destination (Arrow)   |
|-----------------------|-----------------------|
| `bool`                | `BOOL`                |
| `(u)int{8,16,32,64}`  | `(U)INT{8,16,32,64}`  |
| `float32`             | `FLOAT`               |
| `float64`             | `DOUBLE`              |
| `str / unicode`       | `STRING`              |
| `pd.Categorical`      | `DICTIONARY`          |
| `pd.Timestamp`        | `TIMESTAMP(unit=ns)`  |
| `datetime.date`       | `DATE`                |
| `datetime.time`       | `TIME64`              |
] ]

.pull-right[ .small[
| Source (Arrow)                  | Destination (Pandas)                           |
|---------------------------------|------------------------------------------------|
| `BOOL`                          | `bool`                                         |
| `BOOL` with nulls	              | `object` (with values `True`, `False`,`None`)  |
| `(U)INT{8,16,32,64}`            | `(u)int{8,16,32,64}`                           |
| `(U)INT{8,16,32,64}` with nulls	| `float64`                                      |
| `FLOAT`	                        | `float32`                                      |
| `DOUBLE`	                      | `float64`                                      |
| `STRING`	                      | `str`                                          |
| `DICTIONARY`	                  | `pd.Categorical`                               |
| `TIMESTAMP(unit=*)`             | `pd.Timestamp` (`np.datetime64[ns]`)           |
| `DATE`	                        | `object` (with `datetime.date` objects)        |
| `TIME64`	                      | `object` (with `datetime.time` objects)        |
] ]



.footnote[From [Type differences](https://arrow.apache.org/docs/python/pandas.html#type-differences) documentation]

---

## Series & data copies

Due to these discrepancies it is much more likely that converting from Arrow array to a Panda series will require a type to be changed in which case the data will need to be copied. Like `to_numpy()` the `to_pandas()` method also accepts the `zero_copy_only` argument, however it defaults to `False`.

.pull-left[
```{python}
pa.array([1,2,3,4]).to_pandas()
pa.array(["hello", "world"]).to_pandas()
pa.array(["hello", "world"]).dictionary_encode().to_pandas()
```
]

.pull-right[
```{python error=TRUE}
pa.array([1,2,3,4]).to_pandas(zero_copy_only=True)
pa.array(["hello", "world"]).to_pandas(zero_copy_only=True)
pa.array(["hello", "world"]).dictionary_encode().to_pandas(zero_copy_only=True)
```
]

.footnote[Note that Arrow arrays are converted to Series while tables & batches are converted to DataFrames]


---

## Zero Copy Series Conversions

> Zero copy conversions from `Array` or `ChunkedArray` to NumPy arrays or pandas Series are possible in certain narrow cases:
>
> * The Arrow data is stored in an integer (signed or unsigned `int8` through `int64`) or floating point type (`float16` through `float64`). This includes many numeric types as well as timestamps.
>
> * The Arrow data has no null values (since these are represented using bitmaps which are not supported by pandas).
> 
> * For `ChunkedArray`, the data consists of a single chunk, i.e. `arr.num_chunks == 1`. Multiple chunks will always require a copy because of pandas’s contiguousness requirement.
>
> In these scenarios, `to_pandas` or `to_numpy` will be zero copy. In all other scenarios, a copy will be required.

---

## DataFrame & data copies

```{python error=TRUE}
table.to_pandas()
table.to_pandas(zero_copy_only=True)
table.drop(['name']).to_pandas(zero_copy_only=True)
pa.table([num,year], names=["num","year"]).to_pandas(zero_copy_only=True)
```

.footnote[[Source](https://arrow.apache.org/docs/python/pandas.html#zero-copy-series-conversions)]

---

## Pandas -> Arrow

To convert from a Pandas DataFrame to an Arrow Table we can use the `from_pandas()` method (schemas can also be inferred from DataFrames)

```{python}
df = pd.DataFrame({
  'x': np.random.normal(size=5),
  'y': ["A","A","B","C","C"],
  'z': [1,2,3,4,5]
})

pa.Table.from_pandas(df)
pa.Schema.from_pandas(df)
```

.footnote[The import of Pandas indexes is governed by the `preserve_index` argument]

---
class: center, middle

## An aside on tabular file formats

---

## Comma Separated Values

This and other text & delimiter based file formats are the most common and generally considered the most portable, however they have a number of significant draw backs

* no explicit schema or other metadata

* column types must be inferred from the data

* numerical values stored as text (efficiency and precision issues)

* limited compression options

---

## (Apache) Parquet

> ... provides a standardized open-source columnar storage format for use in data analysis systems. It was created originally for use in Apache Hadoop with systems like Apache Drill, Apache Hive, Apache Impala, and Apache Spark adopting it as a shared standard for high performance data IO.


Core features:
> The values in each column are physically stored in contiguous memory locations and this columnar storage provides the following benefits:
>
> * Column-wise compression is efficient and saves storage space
> * Compression techniques specific to a type can be applied as the column values tend to be of the same type
> * Queries that fetch specific column values need not read the entire row data thus improving performance
> * Different encoding techniques can be applied to different columns


---

## Feather

> ... is a portable file format for storing Arrow tables or data frames (from languages like Python or R) that utilizes the Arrow IPC format internally. Feather was created early in the Arrow project as a proof of concept for fast, language-agnostic data frame storage for Python (pandas) and R.

Core features:
* Direct columnar serialization of Arrow tables

* Supports all Arrow data types and compression

* Language agnostic

* Metadata makes it possible to read only the necessary columns for an operation

---
class: middle

.center[
## Example - File Format Performance
]

.footnote[ Based on [Apache Arrow: Read DataFrame With Zero Memory](https://towardsdatascience.com/apache-arrow-read-dataframe-with-zero-memory-69634092b1a) ]
---

## Building a large dataset

```{python}
np.random.seed(1234)

df = (
    pd.read_csv("https://sta663-sp22.github.io/slides/data/penguins.csv")
      .sample(1000000, replace=True)
      .reset_index(drop=True)
)

num_cols = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm",	"body_mass_g"]
df[num_cols] = df[num_cols] + np.random.normal(size=(df.shape[0],len(num_cols)))

df
```

---

## Create output files

```{python eval=!dir.exists("scratch/")}
import os
os.makedirs("scratch/", exist_ok=True)

df.to_csv("scratch/penguins-large.csv")
df.to_parquet("scratch/penguins-large.parquet")

import pyarrow.feather

pyarrow.feather.write_feather(
    pa.Table.from_pandas(df), 
    "scratch/penguins-large.feather"
)

pyarrow.feather.write_feather(
    pa.Table.from_pandas(df.dropna()), 
    "scratch/penguins-large_nona.feather"
)
```

---

## File Sizes

```{python}
def file_size(f):
    x = os.path.getsize(f)
    print(f, "\t\t", round(x / (1024 * 1024),2), "MB")
```

```{python}
file_size( "scratch/penguins-large.csv" )
file_size( "scratch/penguins-large.parquet" )
file_size( "scratch/penguins-large.feather" )
file_size( "scratch/penguins-large_nona.feather" )
```

---

## Read Performance

```python
%timeit pd.read_csv("scratch/penguins-large.csv")
```
```
## 566 ms ± 11.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

```python
%timeit pd.read_parquet("scratch/penguins-large.parquet")
```
```
## 116 ms ± 3.43 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

--

&nbsp;

```python
%timeit pyarrow.csv.read_csv("scratch/penguins-large.csv")
```
```
## 34.2 ms ± 99 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

```python
%timeit pyarrow.parquet.read_table("scratch/penguins-large.parquet")
```
```
## 55.8 ms ± 104 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

--

&nbsp;

```python
%timeit pyarrow.feather.read_table("scratch/penguins-large.feather")
```
```
## 7.12 ms ± 30.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

```python
%timeit pyarrow.feather.read_table("scratch/penguins-large_nona.feather")
```
```
## 7.4 ms ± 263 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

---

## Read Performance (Arrow -> Pandas)

```python
%timeit pyarrow.csv.read_csv("scratch/penguins-large.csv").to_pandas()
```
```
## 88.9 ms ± 1.45 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

--

&nbsp;

```python
%timeit pyarrow.parquet.read_table("scratch/penguins-large.parquet").to_pandas()
```
```
## 109 ms ± 153 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

--

&nbsp;

```python
%timeit pyarrow.feather.read_feather("scratch/penguins-large.feather")
```
```
## 61.5 ms ± 333 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

```python
%timeit pyarrow.feather.read_feather("scratch/penguins-large_nona.feather")
```
```
## 60.6 ms ± 70.6 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

---

## Column subset calculations - CSV & Parquet

```python
%timeit pd.read_csv("scratch/penguins-large.csv")["flipper_length_mm"].mean()
```
```
## 568 ms ± 9.43 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

```python
%timeit pd.read_parquet("scratch/penguins-large.parquet",  columns=["flipper_length_mm"]).mean()
```
```
## 9.74 ms ± 53.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

--

&nbsp;

```python
%timeit pyarrow.parquet.read_table("scratch/penguins-large.parquet", columns=["flipper_length_mm"]).to_pandas().mean()
```
```
## 8.46 ms ± 183 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

```python
%timeit pyarrow.parquet.read_table("scratch/penguins-large.parquet")["flipper_length_mm"].to_pandas().mean()
```
```
## 60.7 ms ± 1.04 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

---

## Column subset calculations - Feather

```python
%timeit pyarrow.feather.read_table("scratch/penguins-large.feather", columns=["flipper_length_mm"]).to_pandas().mean()
```
```
## 8.1 ms ± 53.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

```python
%timeit pyarrow.feather.read_table("scratch/penguins-large.feather")["flipper_length_mm"].to_pandas().mean()
```
```
## 14.4 ms ± 403 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

--

&nbsp;

```python
%timeit pyarrow.feather.read_table("scratch/penguins-large_nona.feather", columns=["flipper_length_mm"]).to_pandas().mean()
```
```
## 4.5 ms ± 76.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

```python
%timeit pyarrow.feather.read_table("scratch/penguins-large_nona.feather")["flipper_length_mm"].to_pandas().mean()
```
```
## 10 ms ± 21.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

---
class: middle

.center[
## Demo 1 - Remote Files + Datasets
]