Problem Description
Now that the categorical values are included in the metadata, it would be helpful to be able to indicate columns that have been detected/assigned the sdtype categorical but also have a very high cardinality.
Expected behavior
Add the high_cardinality as an optional flag to the metadata for ordinal and categorical sdtype columns. The flag must be a boolean. If high_cardinality is set to True, then range_values is not allowed to be set for the column.
During autodetection, if a high cardinality ordinal or categorical column is detected (>500 unique values) then the high_cardinality flag should be set to True and range_values should not be autodetected for the column.
Problem Description
Now that the categorical values are included in the metadata, it would be helpful to be able to indicate columns that have been detected/assigned the sdtype categorical but also have a very high cardinality.
Expected behavior
Add the
high_cardinalityas an optional flag to the metadata forordinalandcategoricalsdtype columns. The flag must be a boolean. Ifhigh_cardinalityis set toTrue, thenrange_valuesis not allowed to be set for the column.During autodetection, if a high cardinality ordinal or categorical column is detected (>500 unique values) then the
high_cardinalityflag should be set toTrueandrange_valuesshould not be autodetected for the column.