Today, data is more valuable than Gold. Many companies have been building revolutionary machine learning and artificial intelligence products for decades now. Algorithms have been used for making our lives easier in countless ways, from identifying disease, to predicting the weather, to even recommending movies. AI is revolutionizing the world and data is the fuel that powers it. Recently, with the release of large language models like ChatGPT, Claude, and Gemini, the world has seen the power of AI and data science. Millions of people now have a seemingly endless amount of information at their fingertips. We wouldn't have any of it without data. Everywhere we look, everywhere we go, everything we see is data. Data is becoming the lifeblood of the world. It is the becoming the foundation of everything we do, from the way we communicate to the way we work, heck even the way we drive. Data is everywhere, it's not going anywhere, and it is growing at an exponential rate. With the rise of big data, data science, and machine learning, the importance of data has never been more obvious. But what is data? To make it as simple as possible, data is information, and information can be anything from numbers to text to images. However, data is more than just information; it is also a type of information. Data types are crucial for understanding how to work with data, and they play a significant role in programming. # Introduction to Data Typing In programming, data types are used to separate different types of data, such as integers, strings, and booleans. It is incredibly important to ensure you are properly typing your variables, as it can have a huge impact on the performance, memory management, and security of your program. Data typing is the process of assigning a type to a variable, which tells the computer how to interpret the data stored in that variable. There are literally dozens of data types spread across dozens of programming languages, but the most common ones are integers, floats, strings, and booleans. An integer is a whole number, a float is a number with a decimal point, a string is a sequence of characters, and a boolean is a value that is either true or false. These data types are the building blocks of programming, and they are used to represent everything from numbers to text to logic. For example: ```python # Integers (whole numbers) age = 25 year = 2024 temperature = -5 # Floats (decimal numbers) height = 5.11 pi = 3.14159 bank_balance = -123.45 # Strings (text) name = "Chris" message = 'I am the master of the universe!' address = "123 Data Science St." # Booleans (True/False) is_student = True has_license = False is_raining = True ``` On the surface learning data types is pretty straight forward, if its a whole number use an integer, if its a decimal use a float, if its text use a string, and if its a true or false statement use a boolean. However, once you get a little bit deeper into programming, you'll realize that data types are much more complex than that. There are many different types of integers, floats, strings, and booleans, and each one has its own unique properties and behaviors. For example, in Python, there are four different types of integers: _int8_, _int16_, _int32_, and _int64_. Each of these types has a different range of values that it can represent, and each one takes up a different amount of memory. Similarly, there are different types of floats, strings, and booleans. Properly typing your data can have massive effects on performance, memory usage, and security. For example, using a smaller integer type like _int8_ instead of _int64_ can save memory and improve performance. Similarly, using a strongly typed array instead of a dynamic array can improve memory usage and reduce the risk of security vulnerabilities. # Good ol' Memory If you've been around computers a long time it's likely you've seen the evolution of memory. From the days of kilobytes to megabytes to gigabytes to terabytes, memory has come a long way. Memory is a crucial aspect of programming, and it is important to understand how data types can impact memory usage. When you create a variable in a program, the computer allocates memory to store the data associated with that variable. The amount of memory allocated depends on the data type of the variable. Modern computers have either 32 bit or 64 bit memory architecture, which means they can store 32 bits or 64 bits of data at a time. This is important because different data types require different amounts of memory to store. For example, an _int8_ variable requires 8 bits of memory, while an _int64_ variable requires 64 bits of memory. This means that an _int8_ variable uses 8 times less memory than an _int64_ variable to store the same value. Depending on your environment, these memory savings can be absolutely crucial. They can be the difference between being able to work with data and not even being able to load it into memory. I've created a couple of tools to help ingrain how memory is stored. Please feel free around to play with this data type visualizer here to help get an understanding of how data types are stored in memory.

Here is also a handy chart to help you understand how much memory different data types use:

# YeeYeeYeeYee Live Action Let's dive into a real world example to see how data typing can directly impact the performance and memory usage of a program. We'll use Python and Pandas to load a data set and optimize the data types to reduce memory usage. We'll also take a look at how different data types are represented in memory and how they can impact the performance of a program. It's important to note that this example is specific to Python and Pandas, but the concepts can be applied to any programming language and data processing library. Also keep in mind that pandas automatically assigns data types to columns when loading data, so it's important to check and optimize the data types to reduce memory usage as Pandas uses default data types such as _int64_ for integers even though they possibly can be optimized with a smaller int like an _int8_. Let's get started! The dataset that will be used for this example is a dataset of NFL player statistics sourced from Kaggle. If you would like to follow along, the dataset can be downloaded from here: [NFL Player Stats](https://www.kaggle.com/datasets/zynicide/nfl-football-player-stats) There is two datasets included, but we will specifically focus on the games. Our dataset contains 46 columns and 1,024,164 rows, so it is fairly large and will be a good example to demonstrate how memory optimization is very important and can lead to extreme cost and time savings. Here is a quick look at the available columns in the dataset: | | | | | | ---------------------- | -------------------------- | ------------------------------- | ------------------------ | | player_id | date | game_number | age | | passing_attempts | passing_completions | passing_yards | passing_rating | | passing_touchdowns | passing_interceptions | passing_sacks | passing_sacks_yards_lost | | rushing_attempts | rushing_yards | rushing_touchdowns | receiving_targets | | receiving_receptions | receiving_yards | receiving_touchdowns | kick_return_attempts | | kick_return_yards | kick_return_touchdowns | punt_return_attempts | punt_return_yards | | punt_return_touchdowns | defense_sacks | defense_tackles | defense_tackle_assists | | defense_interceptions | defense_interception_yards | defense_interception_touchdowns | defense_safeties | | point_after_attemps | point_after_makes | field_goal_attempts | field_goal_makes | | punting_attempts | punting_yards | punting_blocked | team | | game_location | opponent | game_won | year | | player_team_score | opponent_score | | | To get an initial look at the memory usage of the dataset, we can use the following code: ```python nfl_game_stats_df.memory_usage(deep=True).sum() / 1024**2 ``` The dataset is using 525.47 mb of memory initially. This is a fairly large amount of memory, and we can optimize it by converting the data types of the columns to more appropriate types. For example, we can convert the _int64_ columns to _int8_ or _int16_ if the values in the columns will never exceed the maximum value supported by those types. We can also convert the _object_ columns to _category_ if there are a finite number of unique values in the columns. Let's take a deeper look at the data types of the columns in the dataset and their memory usage to see how we can ooptimize them: ```python # Get memory usage of each column and convert to MB from bytes. (deep=True to include object columns) mem_usage = nfl_game_stats_df.memory_usage(deep=True) / 1024**2 # Get data types of each column dtypes = nfl_game_stats_df.dtypes # Split the data into two halves for easier viewing left, right = np.array_split(mem_usage, 2) left = left[1:] left = left.reset_index() left.columns = ["Columns", "Values"] right = right.reset_index() right.columns = ["Columns", "Values"] df1 = pd.DataFrame({ 'Features': left['Columns'], 'Memory (MB)': left['Values'], 'Dtype': [dtypes[col] for col in left['Columns']] }) df2 = pd.DataFrame({ 'Features': right['Columns'], 'Memory (MB)': right['Values'], 'Dtype': [dtypes[col] for col in right['Columns']] }) pd.concat([df1, df2], axis=1) ``` | | Features | Memory (MB) | Dtype | Features | Memory (MB) | Dtype | | --: | :----------------------- | ----------: | :------------- | :------------------------------ | ----------: | :------ | | 0 | player_id | 7.814 | int64 | receiving_receptions | 7.814 | int64 | | 1 | year | 7.814 | int64 | receiving_yards | 7.814 | int64 | | 2 | date | 7.814 | datetime64[ns] | receiving_touchdowns | 7.814 | int64 | | 3 | game_number | 7.814 | int64 | kick_return_attempts | 7.814 | int64 | | 4 | age | 53.717 | object | kick_return_yards | 7.814 | int64 | | 5 | team | 50.789 | object | kick_return_touchdowns | 7.814 | int64 | | 6 | game_location | 48.836 | object | punt_return_attempts | 7.814 | int64 | | 7 | opponent | 50.789 | object | punt_return_yards | 7.814 | int64 | | 8 | game_won | 0.977 | bool | punt_return_touchdowns | 7.814 | int64 | | 9 | player_team_score | 7.814 | int64 | defense_sacks | 7.814 | float64 | | 10 | opponent_score | 7.814 | int64 | defense_tackles | 7.814 | int64 | | 11 | passing_attempts | 7.814 | int64 | defense_tackle_assists | 7.814 | int64 | | 12 | passing_completions | 7.814 | int64 | defense_interceptions | 7.814 | int64 | | 13 | passing_yards | 7.814 | int64 | defense_interception_yards | 7.814 | int64 | | 14 | passing_rating | 7.814 | float64 | defense_interception_touchdowns | 7.814 | int64 | | 15 | passing_touchdowns | 7.814 | int64 | defense_safeties | 7.814 | int64 | | 16 | passing_interceptions | 7.814 | int64 | point_after_attemps | 7.814 | int64 | | 17 | passing_sacks | 7.814 | int64 | point_after_makes | 7.814 | int64 | | 18 | passing_sacks_yards_lost | 7.814 | int64 | field_goal_attempts | 7.814 | int64 | | 19 | rushing_attempts | 7.814 | int64 | field_goal_makes | 7.814 | int64 | | 20 | rushing_yards | 7.814 | int64 | punting_attempts | 7.814 | int64 | | 21 | rushing_touchdowns | 7.814 | int64 | punting_yards | 7.814 | int64 | | 22 | receiving_targets | 7.814 | int64 | punting_blocked | 7.814 | int64 | There are a few things that immediately stand out in the output: 1. The _object_ columns are using a significant amount of memory. 2. The _int64_ columns are using more memory than necessary. 3. All of the 64 bit columns are using the same amount of memory which is a great visual example as to how it's not necessarily the value that changes memory, but the data type. Let's take a look at the object columns first since they are using by far the most amount of memory: ```python nfl_game_stats_df[['age', 'team', 'game_location', 'opponent']].head() ``` | | age | team | game_location | opponent | | --: | :----- | :--- | :------------ | :------- | | 0 | 23-120 | SEA | A | CHI | | 1 | 23-127 | SEA | H | RAI | | 2 | 23-134 | SEA | A | DEN | | 3 | 23-142 | SEA | H | CIN | | 4 | 23-148 | SEA | A | NWE | You might have been thinking why is the age column an object? Shouldn't it be an integer? It's clear in the output that age isn't a whole number and likely represents age with years and days, so the first player is 23 years old, 120 days. This is a perfect example of why it's important to check the data types of your columns and make sure they are appropriate for the data they contain. The team, game_location, and opponent columns are all strings, so they are correctly typed as objects, however they can be better typed as a category. What does a category do? A category is a data type that is used to represent a fixed number of unique values. In simpler terms, if it's a categorical variable, type it as such. It is more memory efficient than an object data type, especially when the number of unique values is small. A quick way to check is using Pandas .nunique() method: ```python nfl_game_stats_df[['age', 'team', 'game_location', 'opponent']].nunique() ``` | | count | | :------------ | -------: | | age | 8125.000 | | team | 42.000 | | game_location | 3.000 | | opponent | 42.000 | We know that the dataset contains 1,024,164 rows of data and since all four of these columns have a really low number of unique values, we can deduce that these are all categorical variables. We can convert them to a category data type to save memory: ```python memory_data = [] columns = ['age', 'team', 'game_location', 'opponent'] for col in columns: before = nfl_game_stats_df[col].memory_usage(deep=True) / 1024**2 after = nfl_game_stats_df[col].astype("category").memory_usage(deep=True) / 1024**2 memory_data.append({ 'column': col, 'before_mb': before, 'after_mb': after, 'mb_saved': before - after }) memory_df = pd.DataFrame(memory_data).set_index('column').rename_axis(None) memory_df.loc['total'] = [memory_df['before_mb'].sum(), memory_df['after_mb'].sum(), memory_df['mb_saved'].sum()] ``` | | before_mb | after_mb | mb_saved | | :------------ | --------: | -------: | -------: | | age | 53.718 | 2.632 | 51.086 | | team | 50.790 | 0.980 | 49.810 | | game_location | 48.836 | 0.977 | 47.859 | | opponent | 50.790 | 0.980 | 49.810 | | total | 204.133 | 5.569 | 198.564 | By making this one simple change, we were able to save 198.56 MB of memory, which is a significant amount considering the dataset is only 525.47 MB. That's nearly 38% of the total memory usage of the dataset saved! Strings/objects can really be detrimental to memory usage, so it's important to handle them appropriately. We were able to bring the total memory usage down to 326.9 MB, which is a huge improvement. Next, let's take a look at the integer columns and see if we can optimize them as well. We can use the same approach as before to check the memory usage of the integer columns and see if we can convert them to a smaller data type. I wrote a function that checks the min and max of each integer column and determines the optimal integer type based on the range of values: ```python def optimize_ints_report(df): memory_data = [] int_cols = df.select_dtypes(include=['int64']).columns for col in int_cols: col_min = df[col].min() col_max = df[col].max() before = df[col].memory_usage(deep=True) / 1024**2 # Determine optimal type if col_min >= 0: dtype = ('uint8' if col_max < 256 else 'uint16' if col_max < 65536 else 'uint32' if col_max < 4294967296 else 'uint64') else: dtype = ('int8' if -128 <= col_min and col_max < 128 else 'int16' if -32768 <= col_min and col_max < 32768 else 'int32' if -2147483648 <= col_min and col_max < 2147483648 else 'int64') after = df[col].astype(dtype).memory_usage(deep=True) / 1024**2 memory_data.append({ 'column': col, 'before_mb': before, 'after_mb': after, 'mb_saved': before - after, 'optimal_type': dtype }) memory_df = pd.DataFrame(memory_data).set_index('column').rename_axis(None) memory_df.loc['total'] = [memory_df['before_mb'].sum(), memory_df['after_mb'].sum(), memory_df['mb_saved'].sum(), ''] return memory_df ``` | | before_mb | after_mb | mb_saved | optimal_type | | :------------------------------ | --------: | -------: | -------: | :----------- | | player_id | 7.814 | 1.954 | 5.860 | uint16 | | year | 7.814 | 1.954 | 5.860 | uint16 | | game_number | 7.814 | 0.977 | 6.837 | uint8 | | player_team_score | 7.814 | 0.977 | 6.837 | uint8 | | opponent_score | 7.814 | 0.977 | 6.837 | uint8 | | passing_attempts | 7.814 | 0.977 | 6.837 | uint8 | | passing_completions | 7.814 | 0.977 | 6.837 | uint8 | | passing_yards | 7.814 | 1.954 | 5.860 | int16 | | passing_touchdowns | 7.814 | 0.977 | 6.837 | uint8 | | passing_interceptions | 7.814 | 0.977 | 6.837 | uint8 | | passing_sacks | 7.814 | 0.977 | 6.837 | uint8 | | passing_sacks_yards_lost | 7.814 | 0.977 | 6.837 | uint8 | | rushing_attempts | 7.814 | 0.977 | 6.837 | uint8 | | rushing_yards | 7.814 | 1.954 | 5.860 | int16 | | rushing_touchdowns | 7.814 | 0.977 | 6.837 | uint8 | | receiving_targets | 7.814 | 0.977 | 6.837 | uint8 | | receiving_receptions | 7.814 | 0.977 | 6.837 | uint8 | | receiving_yards | 7.814 | 1.954 | 5.860 | int16 | | receiving_touchdowns | 7.814 | 0.977 | 6.837 | uint8 | | kick_return_attempts | 7.814 | 0.977 | 6.837 | uint8 | | kick_return_yards | 7.814 | 1.954 | 5.860 | int16 | | kick_return_touchdowns | 7.814 | 0.977 | 6.837 | uint8 | | punt_return_attempts | 7.814 | 0.977 | 6.837 | uint8 | | punt_return_yards | 7.814 | 1.954 | 5.860 | int16 | | punt_return_touchdowns | 7.814 | 0.977 | 6.837 | uint8 | | defense_tackles | 7.814 | 0.977 | 6.837 | uint8 | | defense_tackle_assists | 7.814 | 0.977 | 6.837 | uint8 | | defense_interceptions | 7.814 | 0.977 | 6.837 | uint8 | | defense_interception_yards | 7.814 | 1.954 | 5.860 | int16 | | defense_interception_touchdowns | 7.814 | 0.977 | 6.837 | uint8 | | defense_safeties | 7.814 | 0.977 | 6.837 | uint8 | | point_after_attemps | 7.814 | 0.977 | 6.837 | uint8 | | point_after_makes | 7.814 | 0.977 | 6.837 | uint8 | | field_goal_attempts | 7.814 | 0.977 | 6.837 | uint8 | | field_goal_makes | 7.814 | 0.977 | 6.837 | uint8 | | punting_attempts | 7.814 | 0.977 | 6.837 | uint8 | | punting_yards | 7.814 | 1.954 | 5.860 | int16 | | punting_blocked | 7.814 | 0.977 | 6.837 | uint8 | | total | 296.927 | 45.911 | 251.017 | | It's like magic, really. By converting the integer columns to their optimal integer types, we were able to save an additional 251.02 MB of memory, bringing the total memory usage of the dataset down to 75.89 MB. That's a 85.6% reduction in memory usage, which is absolutely incredible. To test out some of these memory savings we will put the dataset through a gauntlet of tests, to compare the unoptimized dataset with the optimized dataset. We will test the time it takes to perform value counts, complex groupby operations, string operations, and complex filtering on the dataset. ```python def benchmark_extended_ops(): # Value counts start = time.time() nfl_game_stats_df['team'].value_counts() t1 = time.time() - start start = time.time() nfl_game_stats_df_optimized['team'].value_counts() t2 = time.time() - start print(f"Value counts - Original: {t1:.2f}s, Optimized: {t2:.2f}s") # Multiple groupby operations start = time.time() nfl_game_stats_df.groupby(['team', 'year'])['passing_yards'].mean() t1 = time.time() - start start = time.time() nfl_game_stats_df_optimized.groupby(['team', 'year'], observed=True)['passing_yards'].mean() t2 = time.time() - start print(f"Complex groupby - Original: {t1:.2f}s, Optimized: {t2:.2f}s") # String operations start = time.time() nfl_game_stats_df['team'].str.contains('A') t1 = time.time() - start start = time.time() nfl_game_stats_df_optimized['team'].str.contains('A') t2 = time.time() - start print(f"String ops - Original: {t1:.2f}s, Optimized: {t2:.2f}s") # Complex filtering start = time.time() nfl_game_stats_df[(nfl_game_stats_df['passing_yards'] > 200) & (nfl_game_stats_df['rushing_yards'] > 50)] t1 = time.time() - start start = time.time() nfl_game_stats_df_optimized[(nfl_game_stats_df_optimized['passing_yards'] > 200) & (nfl_game_stats_df_optimized['rushing_yards'] > 50)] t2 = time.time() - start print(f"Complex filter - Original: {t1:.2f}s, Optimized: {t2:.2f}s") benchmark_extended_ops() ``` | Operation | Original (s) | Optimized (s) | | --------------- | ------------ | ------------- | | Value counts | 0.13 | 0.00 | | Complex groupby | 0.12 | 0.02 | | String ops | 0.24 | 0.00 | | Complex filter | 0.02 | 0.00 | The proof is in the pudding, by optimizing the data types of the columns in the dataset, we were able to significantly reduce the time it takes to perform operations on the data. This is because smaller data types are faster to process and require less memory to store, which leads to faster execution times. When working with huge data and/or having to perform a lot of operations these time savings add up very quickly. # All Good Things Come To An End In conclusion, data typing is a crucial aspect of programming that can have a huge impact on the performance, memory usage, and security of your program. By properly typing your data, you can optimize your program to run faster, use less memory, and be more secure. It's important to understand the different data types available to you and how they can be used to represent your data. By taking the time to optimize the data types of your columns and projects in general, you can save a significant amount of memory and improve the performance of your program. Remember, data is the lifeblood of the world, and data typing is the key to unlocking its full potential.

Am I Your Type?