How can one append to parquet files and how does it affect partitioning? - parquet

Question

Does parquet allow appending to a parquet file periodically ?
How does appending relate to partitioning if any ? For example if i was able to identify a column that had low cardinality and partition it by that column, if i were to append more data to it would parquet be able to automatically append data while preserving partitioning or would one have to repartition the file ?

Answer 1

Does parquet allow appending to a parquet file periodically ?
Yes and No. The parquet spec describes a format that could be appended to by reading the existing footer, writing a row group, and then writing out a modified footer. This process is described a bit here.
Not all implementations support this operation. The only implementation I am aware of at the moment is fastparquet (see this answer). It is usually acceptable, less complexity, and potentially better performance to cache and batch, either by caching in memory or writing the small files and batching them together at some point later.
How does appending relate to partitioning if any?
Parquet does not have any concept of partitioning.
Many tools that support parquet implement partitioning. For example, pyarrow has a datasets feature which supports partitioning. If you were to append new data using this feature a new file would be created in the appropriate partition directory.

Answer 2

Its possible to append row groups to already existing parquet file using fastparquet.
Here is my SO answer on the same topic.
From fast parquet docs
append: bool (False) or ‘overwrite’ If False, construct data-set from
scratch; if True, add new row-group(s) to existing data-set. In the
latter case, the data-set must exist, and the schema must match the
input data.
from fastparquet import write
write('output.parquet', df, append=True)
EXAMPLE UPDATE:
Here is a PY script. The first run, it will create a file with one row group. Subsequent runs, it will append row groups to the same parquet file.
import os.path
import pandas as pd
from fastparquet import write
df = pd.DataFrame(data={'col1': [1, 2,], 'col2': [3, 4]})
file_path = "C:\\Users\\nsuser\\dev\\write_parq_row_group.parquet"
if not os.path.isfile(file_path):
write(file_path, df)
else:
write(file_path, df, append=True)