vaex

Out-of-Core DataFrames to visualize and explore big tabular datasets

MIT 59 个版本

Maarten A. Breddels <maartenbreddels@gmail.com>

安装

pip install vaex

poetry add vaex

pipenv install vaex

conda install vaex

描述

What is Vaex?

Vaex is a high performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular datasets. It calculates statistics such as mean, sum, count, standard deviation etc, on an N-dimensional grid for more than a billion (10^9) samples/rows per second. Visualization is done using histograms, density plots and 3d volume rendering, allowing interactive exploration of big data. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted).

Installing

With pip:

$ pip install vaex

Or conda:

$ conda install -c conda-forge vaex

For more details, see the documentation

Key features

Instant opening of Huge data files (memory mapping)

HDF5 and Apache Arrow supported.

opening1a

opening1b

Read the documentation on how to efficiently convert your data from CSV files, Pandas DataFrames, or other sources.

Lazy streaming from S3 supported in combination with memory mapping.

opening1c

Expression system

Don't waste memory or time with feature engineering, we (lazily) transform your data when needed.

expression

Out-of-core DataFrame

Filtering and evaluating expressions will not waste memory by making copies; the data is kept untouched on disk, and will be streamed only when needed. Delay the time before you need a cluster.

occ-animated

Fast groupby / aggregations

Vaex implements parallelized, highly performant groupby operations, especially when using categories (>1 billion/second).

groupby

Fast and efficient join

Vaex doesn't copy/materialize the 'right' table when joining, saving gigabytes of memory. With subsecond joining on a billion rows, it's pretty fast!

join

More features

Remote DataFrames (documentation coming soon)
Integration into Jupyter and Voila for interactive notebooks and dashboards
Machine Learning without (explicit) pipelines

Contributing

See contributing page.

Slack

Join the discussion in our Slack channel!

Learn more about Vaex

Articles
Follow our tutorials
Watch our more recent talks:
- PyData London 2019
- SciPy 2019
Contact us for data science solutions, training, or enterprise support at https://vaex.io/

版本列表

4.19.0 2026-02-03

4.17.0 2023-07-21

4.16.0 2022-12-02

4.15.0 2022-11-23

4.14.0 2022-10-11

4.13.0 2022-09-23

4.12.0 2022-08-26

4.11.1 2022-07-26

4.11.0 2022-07-26

4.9.2 2022-06-03

4.9.1 2022-04-12

4.9.0 2022-04-11

4.8.0 2022-02-07

4.7.0 2021-12-18

4.6.0 2021-11-26

4.5.0 2021-09-10

4.4.0 2021-08-13

4.3.0 2021-06-17

4.2.0 2021-06-04

4.1.0 2021-03-18

4.0.1 2021-03-18

4.0.0 2021-03-09

4.0.0a9 2021-01-19

4.0.0a8 2021-01-13

4.0.0a7 2021-01-08

4.0.0a6 2020-12-08

4.0.0a5 2020-12-02

4.0.0a4 2020-11-25

4.0.0a3 2020-11-24

4.0.0a2 2020-11-23

4.0.0a13 2021-01-27

4.0.0a12 2021-01-26

4.0.0a11 2021-01-26

4.0.0a10 2021-01-22

4.0.0a1 2020-11-23

3.0.0 2020-05-25

2.6.1 2020-01-21

2.6.0 2020-01-21

2.5.0 2019-12-16

2.4.0 2019-11-27

2.3.0 2019-10-15

2.2.1 2019-09-27

2.2.0 2019-09-27

2.1.0 2019-08-26

2.0.2 2019-07-19

2.0.1 2019-07-11

2.0.0 2019-07-06

1.0.1 2019-04-11

1.0.0 2019-04-11

1.0.0b8 2017-12-15

1.0.0b7 2017-12-15

1.0.0b6 2017-04-01

1.0.0b5 2017-03-30

1.0.0b4 2016-10-26

1.0.0b3 2016-10-15

1.0.0b2 2016-09-15

1.0.0b1 2016-06-03

0.3.10 2015-03-02

0.3.6 2015-02-26