Skip to content

[SPARK-59271][PYTHON][TESTS] Add tests for pandas.api.types is_* predicates #58537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Spenserrrr wants to merge 1 commit into
base: master
Choose a base branch
from
+507 −0
Open

[SPARK-59271][PYTHON][TESTS] Add tests for pandas.api.types is_* predicates #58537

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions dev/sparktestsupport/modules.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -615,6 +615,7 @@ def __hash__(self):
"pyspark.tests.test_stage_sched",
"pyspark.tests.test_zero_copy_byte_stream",
# unittests for upstream projects
"pyspark.tests.upstream.pandas.test_pandas_api_types",
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_cast",
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_from_pandas_default",
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_from_pandas_non_default",
Expand Down
16 changes: 16 additions & 0 deletions python/pyspark/tests/upstream/pandas/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
32 changes: 32 additions & 0 deletions python/pyspark/tests/upstream/pandas/golden_pandas_api_types_dtype.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
dtype \ predicate input is_bool_dtype is_integer_dtype is_float_dtype is_numeric_dtype is_object_dtype is_datetime64_dtype is_timedelta64_dtype
bool:dtype dtype(bool) True False False True False False False
boolean:dtype dtype(boolean) True False False True False False False
int64:dtype dtype(int64) False True False True False False False
Int64:dtype dtype(Int64) False True False True False False False
int64[pyarrow]:dtype dtype(int64[pyarrow]) False True False True False False False
float64:dtype dtype(float64) False False True True False False False
Float64:dtype dtype(Float64) False False True True False False False
object:dtype dtype(object) False False False False True False False
string[python]:dtype dtype(string[python]) False False False False False False False
string[pyarrow]:dtype dtype(string[pyarrow]) False False False False False False False
category:dtype dtype(category) False False False False False False False
datetime64[ns]:dtype dtype(datetime64[ns]) False False False False False True False
datetime64[us]:dtype dtype(datetime64[us]) False False False False False True False
datetime64[ns,UTC]:dtype dtype(datetime64[ns, UTC]) False False False False False False False
timestamp[us][pyarrow]:dtype dtype(timestamp[us][pyarrow]) False False False False False False False
timedelta64[ns]:dtype dtype(timedelta64[ns]) False False False False False False True
duration[us][pyarrow]:dtype dtype(duration[us][pyarrow]) False False False False False False False
inferred(str):dtype dtype(object) False False False False True False False
inferred(datetime):dtype dtype(datetime64[ns]) False False False False False True False
inferred(timedelta):dtype dtype(timedelta64[ns]) False False False False False False True
int64:series series(int64) False True False True False False False
int64:ndarray ndarray(int64) False True False True False False False
int64:carrier carrier(int64) False True False True False False False
object:series series(object) False False False False True False False
object:ndarray ndarray(object) False False False False True False False
object:carrier carrier(object) False False False False True False False
int64[pyarrow]:series series(int64[pyarrow]) False True False True False False False
int64[pyarrow]:carrier carrier(int64[pyarrow]) False True False True False False False
datetime64[ns,UTC]:series series(datetime64[ns, UTC]) False False False False False False False
datetime64[ns,UTC]:carrier carrier(datetime64[ns, UTC]) False False False False False False False
no-dtype:object other(object) False False False False False False False
33 changes: 33 additions & 0 deletions python/pyspark/tests/upstream/pandas/golden_pandas_api_types_dtype.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
| dtype \ predicate | input | is_bool_dtype | is_integer_dtype | is_float_dtype | is_numeric_dtype | is_object_dtype | is_datetime64_dtype | is_timedelta64_dtype |
|------------------------------|-------------------------------|-----------------|--------------------|------------------|--------------------|-------------------|-----------------------|------------------------|
| bool:dtype | dtype(bool) | True | False | False | True | False | False | False |
| boolean:dtype | dtype(boolean) | True | False | False | True | False | False | False |
| int64:dtype | dtype(int64) | False | True | False | True | False | False | False |
| Int64:dtype | dtype(Int64) | False | True | False | True | False | False | False |
| int64[pyarrow]:dtype | dtype(int64[pyarrow]) | False | True | False | True | False | False | False |
| float64:dtype | dtype(float64) | False | False | True | True | False | False | False |
| Float64:dtype | dtype(Float64) | False | False | True | True | False | False | False |
| object:dtype | dtype(object) | False | False | False | False | True | False | False |
| string[python]:dtype | dtype(string[python]) | False | False | False | False | False | False | False |
| string[pyarrow]:dtype | dtype(string[pyarrow]) | False | False | False | False | False | False | False |
| category:dtype | dtype(category) | False | False | False | False | False | False | False |
| datetime64[ns]:dtype | dtype(datetime64[ns]) | False | False | False | False | False | True | False |
| datetime64[us]:dtype | dtype(datetime64[us]) | False | False | False | False | False | True | False |
| datetime64[ns,UTC]:dtype | dtype(datetime64[ns, UTC]) | False | False | False | False | False | False | False |
| timestamp[us][pyarrow]:dtype | dtype(timestamp[us][pyarrow]) | False | False | False | False | False | False | False |
| timedelta64[ns]:dtype | dtype(timedelta64[ns]) | False | False | False | False | False | False | True |
| duration[us][pyarrow]:dtype | dtype(duration[us][pyarrow]) | False | False | False | False | False | False | False |
| inferred(str):dtype | dtype(object) | False | False | False | False | True | False | False |
| inferred(datetime):dtype | dtype(datetime64[ns]) | False | False | False | False | False | True | False |
| inferred(timedelta):dtype | dtype(timedelta64[ns]) | False | False | False | False | False | False | True |
| int64:series | series(int64) | False | True | False | True | False | False | False |
| int64:ndarray | ndarray(int64) | False | True | False | True | False | False | False |
| int64:carrier | carrier(int64) | False | True | False | True | False | False | False |
| object:series | series(object) | False | False | False | False | True | False | False |
| object:ndarray | ndarray(object) | False | False | False | False | True | False | False |
| object:carrier | carrier(object) | False | False | False | False | True | False | False |
| int64[pyarrow]:series | series(int64[pyarrow]) | False | True | False | True | False | False | False |
| int64[pyarrow]:carrier | carrier(int64[pyarrow]) | False | True | False | True | False | False | False |
| datetime64[ns,UTC]:series | series(datetime64[ns, UTC]) | False | False | False | False | False | False | False |
| datetime64[ns,UTC]:carrier | carrier(datetime64[ns, UTC]) | False | False | False | False | False | False | False |
| no-dtype:object | other(object) | False | False | False | False | False | False | False |
27 changes: 27 additions & 0 deletions python/pyspark/tests/upstream/pandas/golden_pandas_api_types_object.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
object \ predicate input type is_list_like is_hashable is_dict_like is_scalar is_number is_integer is_sequence
int 5 int False True False True True True False
np.int64(5) int64 False True False True True True False
float 5.0 float False True False True True False False
bool True bool False True False True True False False
None NoneType False True False True False False False
np.nan float False True False True True False False
pd.NA NAType False True False True False False False
pd.NaT NaTType False True False True False False False
str 'abc' str False True False True False False False
bytes b'x' bytes False True False True False False False
list [1,2] list True False False False False False True
tuple (1,) tuple True True False False False False True
label ('a','b') tuple True True False False False False True
tuple ([1],) tuple True False False False False False True
dict {'a':1} dict True False True False False False True
set {1,2} set True False False False False False True
generator generator True True False False False False False
range(3) range True True False False False False True
slice(1,2) slice False False False False False False False
np.ndarray([1,2]) ndarray True False False False False False True
pd.Series([1]) Series True False True False False False True
pd.DataFrame DataFrame True False True False False False True
pd.Index([1]) Index True False False False False False True
pd.Categorical Categorical True False False False False False True
np.dtype('int64') Int64DType False True False False False False False
callable len builtin_function_or_method False True False False False False False
28 changes: 28 additions & 0 deletions python/pyspark/tests/upstream/pandas/golden_pandas_api_types_object.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
| object \ predicate | input type | is_list_like | is_hashable | is_dict_like | is_scalar | is_number | is_integer | is_sequence |
|----------------------|----------------------------|----------------|---------------|----------------|-------------|-------------|--------------|---------------|
| int 5 | int | False | True | False | True | True | True | False |
| np.int64(5) | int64 | False | True | False | True | True | True | False |
| float 5.0 | float | False | True | False | True | True | False | False |
| bool True | bool | False | True | False | True | True | False | False |
| None | NoneType | False | True | False | True | False | False | False |
| np.nan | float | False | True | False | True | True | False | False |
| pd.NA | NAType | False | True | False | True | False | False | False |
| pd.NaT | NaTType | False | True | False | True | False | False | False |
| str 'abc' | str | False | True | False | True | False | False | False |
| bytes b'x' | bytes | False | True | False | True | False | False | False |
| list [1,2] | list | True | False | False | False | False | False | True |
| tuple (1,) | tuple | True | True | False | False | False | False | True |
| label ('a','b') | tuple | True | True | False | False | False | False | True |
| tuple ([1],) | tuple | True | False | False | False | False | False | True |
| dict {'a':1} | dict | True | False | True | False | False | False | True |
| set {1,2} | set | True | False | False | False | False | False | True |
| generator | generator | True | True | False | False | False | False | False |
| range(3) | range | True | True | False | False | False | False | True |
| slice(1,2) | slice | False | False | False | False | False | False | False |
| np.ndarray([1,2]) | ndarray | True | False | False | False | False | False | True |
| pd.Series([1]) | Series | True | False | True | False | False | False | True |
| pd.DataFrame | DataFrame | True | False | True | False | False | False | True |
| pd.Index([1]) | Index | True | False | False | False | False | False | True |
| pd.Categorical | Categorical | True | False | False | False | False | False | True |
| np.dtype('int64') | Int64DType | False | True | False | False | False | False | False |
| callable len | builtin_function_or_method | False | True | False | False | False | False | False |
Loading