You can read and write Parquet files with Python using the pyarrow package<\/a>.<\/p>\n Let’s learn how that works now!<\/p>\n The first step is to make sure you have everything you need. In addition to the Python programming language, you will also need pyarrow<\/strong> and the pandas<\/strong> package. You will use pandas because it is another Python package that uses columns as a data format and works well with Parquet files.<\/p>\n You can use pip to install both of these packages. Open up your terminal and run the following command:<\/p>\n If you use Anaconda, you’ll want to install pyarrow using this command instead.<\/p>\n Anaconda should already include pandas, but if not, you can use the same command above by replacing pyarrow with pandas.<\/p>\n Now that you have pyarrow and pandas installed, you can use it to read and write Parquet files!<\/p>\n Writing Parquet files with Python is pretty straightforward. The code to turn a pandas DataFrame into a Parquet file is about ten lines.<\/p>\n Open up your favorite Python IDE or text editor and create a new file. You can name it something like For this example, you have three imports:<\/p>\n The\u00a0\u00a0write_parquet()<\/strong> function takes in a pandas DataFrame and the file name or path to save the Parquet file to. Then, you transform the DataFrame into a pyarrow Table object before converting that into a Parquet File using the Now you are ready to read that file you just created!<\/p>\n Reading the Parquet file you created earlier with Python is even easier. You’ll need about half as many lines of code!<\/p>\n You can put the following code into a new file called something like In this example, you read the Parquet file into a pyarrow Table format and then convert it to a pandas DataFrame using the Table’s to_pandas()<\/strong> method.<\/p>\n When you print out the contents of the DataFrame, you will see the following:<\/p>\n You can see from the output above that the DataFrame contains all data you saved.<\/p>\n One of the strengths of using a Parquet file is that you can read just parts of the file instead of the whole thing. For example, you can read in just some of the columns rather then the whole file!<\/p>\n Here’s an example of how that works:<\/p>\n To read in just the “Languages” and “Users” columns from the Parquet file, you pass in the a list that contains just those column names. Then when you call\u00a0read_table()<\/strong> you pass in the columns you want to read.<\/p>\n Here’s the output when you run this code:<\/p>\n This outputs the pyarrow Table format, which differs slightly from a pandas DataFrame. It tells you information about the different columns; for example, Languages are strings, and Users are of type int64.<\/p>\n If you prefer to work only with pandas DataFrames, the pyarrow package allows that too. As long as you know the Parquet file contains pandas DataFrames, you can use\u00a0read_pandas()<\/strong> instead of\u00a0read_table().<\/strong><\/p>\n Here’s a code example:<\/p>\n When you run this example, the output is a DataFrame that contains just the columns you asked for:<\/p>\n One advantage of using the read_pandas() and to_pandas() methods is that they will maintain any additional index column data in the DataFrame,\u00a0while the pyarrow Table may not.<\/p>\n You can also get the metadata from a Parquet file using Python. Getting the metadata can be useful when you need to inspect an unfamiliar Parquet file to see what type(s) of data it contains.<\/p>\n Here’s a small code snippet that will read the Parquet file’s metadata and schema:<\/p>\n There are two ways to get the Parquet file’s metadata:<\/p>\n The benefit of the former method is that you can also access the\u00a0schema<\/strong> property of the ParquetFile<\/strong> object.<\/p>\n When you run this code, you will see this output:<\/p>\n Nice! You can read the output above to learn the number of rows and columns of data and the size of the data. The schema tells you what the field types are.<\/p>\n Parquet files are becoming more popular in big data and data science-related fields. Python’s pyarrow package makes working with Parquet files easy. You should spend some time experimenting with the code in this tutorial and using it for some of your own Parquet files.<\/p>\n When you want to learn more, check out the Parquet documentation<\/a>.<\/p>\n The post How to Read and Write Parquet Files with Python<\/a> appeared first on Mouse Vs Python<\/a>.<\/p>\n\n Apache Parquet files are a popular columnar storage format used by data scientists and anyone using the Hadoop ecosystem. It was developed to be very efficient in terms of compression and encoding. Check out their documentation if you want to know all the details about how Parquet files work. You can read and write Parquet [\u2026]<\/p>\n The post How to Read and Write Parquet Files with Python<\/a> appeared first on Mouse Vs Python<\/a>.<\/p>\n<\/div>","protected":false},"author":2018,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"slim_seo":{"title":"How to Read and Write Parquet Files with Python - ITTeacherITFreelance.hk","description":"Apache Parquet files are a popular columnar storage format used by data scientists and anyone using the Hadoop ecosystem. It was developed to be very efficient"},"footnotes":""},"categories":[10700],"tags":[],"_links":{"self":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/329587"}],"collection":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/users\/2018"}],"replies":[{"embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/comments?post=329587"}],"version-history":[{"count":1,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/329587\/revisions"}],"predecessor-version":[{"id":329588,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/329587\/revisions\/329588"}],"wp:attachment":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/media?parent=329587"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/categories?post=329587"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/tags?post=329587"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Installing pyarrow<\/h2>\n
python -m pip install pyarrow pandas<\/pre>\n
conda install -c conda-forge pyarrow<\/pre>\n
Writing Parquet Files with Python<\/h2>\n
parquet_file_writer.py<\/code>or use some other descriptive name. Then enter the following code:<\/p>\nimport pandas as pd\r\nimport pyarrow as pa\r\nimport pyarrow.parquet as pq\r\n\r\n\r\ndef write_parquet(df: pd.DataFrame, filename: str) -> None:\r\n table = pa.Table.from_pandas(df)\r\n pq.write_table(table, filename)\r\n \r\n\r\nif __name__ == \"__main__\":\r\n data = {\"Languages\": [\"Python\", \"Ruby\", \"C++\"],\r\n \"Users\": [10000, 5000, 8000],\r\n \"Dynamic\": [True, True, False],\r\n }\r\n df = pd.DataFrame(data=data, index=list(range(1, 4)))\r\n write_parquet(df, \"languages.parquet\")<\/pre>\n\n
pandas<\/code>, so you can create a DataFrame<\/code><\/li>\npyarrow<\/code>, to create a special pyarrow.Table<\/code> object<\/li>\npyarrow.parquet<\/code>to transform the table object into a Parquet file<\/li>\n<\/ul>\nwrite_table()<\/code> method, which writes it to disk.<\/p>\nReading Parquet Files with Python<\/h2>\n
parquet_file_reader.py<\/code> if you want to:<\/p>\nimport pyarrow.parquet as pq\r\n\r\ndef read_parquet(filename: str) -> None:\r\n table = pq.read_table(filename)\r\n df = table.to_pandas()\r\n print(df)\r\n\r\nif __name__ == \"__main__\": \r\n read_parquet(\"languages.parquet\")<\/pre>\n
Languages Users Dynamic\r\n1 Python 10000 True\r\n2 Ruby 5000 True\r\n3 C++ 8000 False<\/pre>\n
import pyarrow.parquet as pq\r\n\r\ndef read_columns(filename: str, columns: list[str]) -> None:\r\n table = pq.read_table(filename, columns=columns)\r\n print(table)\r\n\r\nif __name__ == \"__main__\":\r\n read_columns(\"languages.parquet\", columns=[\"Languages\", \"Users\"])<\/pre>\n
pyarrow.Table\r\nLanguages: string\r\nUsers: int64\r\n----\r\nLanguages: [[\"Python\",\"Ruby\",\"C++\"]]\r\nUsers: [[10000,5000,8000]]\r\n<\/pre>\n
import pyarrow.parquet as pq\r\n\r\ndef read_columns_pandas(filename: str, columns: list[str]) -> None:\r\n table = pq.read_pandas(filename, columns=columns)\r\n df = table.to_pandas()\r\n print(df)\r\n\r\nif __name__ == \"__main__\":\r\n read_columns_pandas(\"languages.parquet\", columns=[\"Languages\", \"Users\"])<\/pre>\n
Languages Users\r\n1 Python 10000\r\n2 Ruby 5000\r\n3 C++ 8000<\/pre>\n
Reading Parquet File Metadata<\/h2>\n
import pyarrow.parquet as pq\r\n\r\ndef read_metadata(filename: str) -> None:\r\n parquet_file = pq.ParquetFile(filename)\r\n metadata = parquet_file.metadata\r\n print(metadata)\r\n print(f\"Parquet file: {filename} Schema\")\r\n print(parquet_file.schema)\r\n\r\nif __name__ == \"__main__\":\r\n read_metadata(\"languages.parquet\")<\/pre>\n\n
<pyarrow._parquet.FileMetaData object at 0x000002312C1355D0>\r\n created_by: parquet-cpp-arrow version 15.0.2\r\n num_columns: 4\r\n num_rows: 3\r\n num_row_groups: 1\r\n format_version: 2.6\r\n serialized_size: 2682\r\nParquet file: languages.parquet Schema\r\n<pyarrow._parquet.ParquetSchema object at 0x000002312BBFDF00>\r\nrequired group field_id=-1 schema {\r\n optional binary field_id=-1 Languages (String);\r\n optional int64 field_id=-1 Users;\r\n optional boolean field_id=-1 Dynamic;\r\n optional int64 field_id=-1 __index_level_0__;\r\n}<\/pre>\nWrapping Up<\/h2>\n
![\\\"IT\u96fb\u8166\u88dc\u7fd2](\\\"http:\/\/itteacheritfreelance.hk\/test\/wordpress\/wp-content\/uploads\/2016\/05\/logo2-2.png\\\"){kind=logo}
\r\nwww.itsec.hk<\/a>
\r\nwww.itsec.vip<\/a>
\r\nwww.itseceu.uk<\/a>
<\/p>","protected":false},"excerpt":{"rendered":"