Interacting with data in binary format
by Sebastian Raschka, Robert Layton, Martin Czygan, Phuong Vo.T.H, Fabrizio Romano,
Python: Real-World Data Science
- Data Science with Python
- Table of Contents
- Data Science with Python
- Meet Your Course Guide
- What's so cool about Data Science?
- Course Structure
- Course Journey
- The Course Roadmap and Timeline
- 1. Course Module 1: Python Fundamentals
- 1. Introduction and First Steps – Take a Deep Breath
- A proper introduction
- Enter the Python
- About Python
- What are the drawbacks?
- Who is using Python today?
- Setting up the environment
- What you need for this course
- How you can run a Python program
- How is Python code organized
- Python's execution model
- Guidelines on how to write good code
- The Python culture
- A note on the IDEs
- 2. Object-oriented Design
- 3. Objects in Python
- 4. When Objects Are Alike
- 5. Expecting the Unexpected
- 6. When to Use Object-oriented Programming
- 7. Python Data Structures
- 8. Python Object-oriented Shortcuts
- 9. Strings and Serialization
- 10. The Iterator Pattern
- 11. Python Design Patterns I
- 12. Python Design Patterns II
- 13. Testing Object-oriented Programs
- 14. Concurrency
- 1. Introduction and First Steps – Take a Deep Breath
- 2. Course Module 2: Data Analysis
- 1. Introducing Data Analysis and Libraries
- 2. NumPy Arrays and Vectorized Computation
- 3. Data Analysis with pandas
- 4. Data Visualization
- 5. Time Series
- 6. Interacting with Databases
- 7. Data Analysis Application Examples
- 3. Course Module 3: Data Mining
- 1. Getting Started with Data Mining
- 2. Classifying with scikit-learn Estimators
- 3. Predicting Sports Winners with Decision Trees
- 4. Recommending Movies Using Affinity Analysis
- 5. Extracting Features with Transformers
- 6. Social Media Insight Using Naive Bayes
- 7. Discovering Accounts to Follow Using Graph Mining
- 8. Beating CAPTCHAs with Neural Networks
- 9. Authorship Attribution
- 10. Clustering News Articles
- 11. Classifying Objects in Images Using Deep Learning
- 12. Working with Big Data
- 13. Next Steps…
- Chapter 1 – Getting Started with Data Mining
- Chapter 2 – Classifying with scikit-learn Estimators
- Chapter 3: Predicting Sports Winners with Decision Trees
- Chapter 4 – Recommending Movies Using Affinity Analysis
- Chapter 5 – Extracting Features with Transformers
- Chapter 6 – Social Media Insight Using Naive Bayes
- Chapter 7 – Discovering Accounts to Follow Using Graph Mining
- Chapter 8 – Beating CAPTCHAs with Neural Networks
- Chapter 9 – Authorship Attribution
- Chapter 10 – Clustering News Articles
- Chapter 11 – Classifying Objects in Images Using Deep Learning
- Chapter 12 – Working with Big Data
- More resources
- 4. Course Module 4: Machine Learning
- 1. Giving Computers the Ability to Learn from Data
- How to transform data into knowledge
- The three different types of machine learning
- An introduction to the basic terminology and notations
- A roadmap for building machine learning systems
- Using Python for machine learning
- 2. Training Machine Learning Algorithms for Classification
- 3. A Tour of Machine Learning Classifiers Using scikit-learn
- 4. Building Good Training Sets – Data Preprocessing
- 5. Compressing Data via Dimensionality Reduction
- 6. Learning Best Practices for Model Evaluation and Hyperparameter Tuning
- 7. Combining Different Models for Ensemble Learning
- 8. Predicting Continuous Target Variables with Regression Analysis
- Introducing a simple linear regression model
- Exploring the Housing Dataset
- Implementing an ordinary least squares linear regression model
- Fitting a robust regression model using RANSAC
- Evaluating the performance of linear regression models
- Using regularized methods for regression
- Turning a linear regression model into a curve – polynomial regression
- A. Reflect and Test Yourself! Answers
- Module 2: Data Analysis
- Module 3: Data Mining
- Chapter 1: Getting Started with Data Mining
- Chapter 2: Classifying with scikit-learn Estimators
- Chapter 3: Predicting Sports Winners with Decision Trees
- Chapter 4: Recommending Movies Using Affinity Analysis
- Chapter 5: Extracting Features with Transformers
- Chapter 6: Social Media Insight Using Naive Bayes
- Chapter 7: Discovering Accounts to Follow Using Graph Mining
- Chapter 8: Beating CAPTCHAs with Neural Networks
- Chapter 9: Authorship Attribution
- Chapter 10: Clustering News Articles
- Chapter 11: Classifying Objects in Images Using Deep Learning
- Chapter 12: Working with Big Data
- Module 4: Machine Learning
- Chapter 1: Giving Computers the Ability to Learn from Data
- Chapter 2: Training Machine Learning
- Chapter 3: A Tour of Machine Learning Classifiers Using scikit-learn
- Chapter 4: Building Good Training Sets – Data Preprocessing
- Chapter 5: Compressing Data via Dimensionality Reduction
- Chapter 6: Learning Best Practices for Model Evaluation and Hyperparameter Tuning
- Chapter 7: Combining Different Models for Ensemble Learning
- Chapter 8: Predicting Continuous Target Variables with Regression Analysis
- B. Bibliography
- 1. Giving Computers the Ability to Learn from Data
- Index
We can read and write binary serialization of Python objects with the pickle module, which can be found in the standard library. Object serialization can be useful, if you work with objects that take a long time to create, like some machine learning models. By pickling such objects, subsequent access to this model can be made faster. It also allows you to distribute Python objects in a standardized way.
pandas includes support for pickling out of the box. The relevant methods are the read_pickle() and to_pickle() functions to read and write data from and to files easily. Those methods will write data to disk in the pickle format, which is a convenient short-term storage format:
>>> df_ex3.to_pickle('example_data/ex_06-03.out') >>> pd.read_pickle('example_data/ex_06-03.out') 1 2 3 4 0 Nam 7 1 male hcm Mai 11 1 female hcm Lan 25 3 female hn Hung 42 3 male tn Nghia 26 3 male dn Vinh 39 3 male vl Hong 28 4 female dn
HDF5 is not a database, but a data model and file format. It is suited for write-one, read-many datasets. An HDF5 file includes two kinds of objects: data sets, which are array-like collections of data, and groups, which are folder-like containers what hold data sets and other groups. There are some interfaces for interacting with HDF5 format in Python, such as h5py which uses familiar NumPy and Python constructs, such as dictionaries and NumPy array syntax. With h5py, we have high-level interface to the HDF5 API which helps us to get started. However, in this module, we will introduce another library for this kind of format called PyTables, which works well with pandas objects:
>>> store = pd.HDFStore('hdf5_store.h5') >>> store <class 'pandas.io.pytables.HDFStore'> File path: hdf5_store.h5 Empty
We created an empty HDF5 file, named hdf5_store.h5. Now, we can write data to the file just like adding key-value pairs to a dict:
>>> store['ex3'] = df_ex3 >>> store['name'] = df_ex2[0] >>> store['hometown'] = df_ex3[4] >>> store <class 'pandas.io.pytables.HDFStore'> File path: hdf5_store.h5 /ex3 frame (shape->[7,4]) /hometown series (shape->[1]) /name series (shape->[1])
Objects stored in the HDF5 file can be retrieved by specifying the object keys:
>>> store['name'] 0 Nam 1 Mai 2 Lan 3 Hung 4 Nghia 5 Vinh 6 Hong Name: 0, dtype: object
Once we have finished interacting with the HDF5 file, we close it to release the file handle:
>>> store.close() >>> store <class 'pandas.io.pytables.HDFStore'> File path: hdf5_store.h5 File is CLOSED
There are other supported functions that are useful for working with the HDF5 format. You should explore ,in more detail, two libraries – pytables and h5py – if you need to work with huge quantities of data.
-
No Comment