← Python

Memory Management & Garbage Collection

Python Internals & Performance

← Memory Management & Garbage Collection

Select a question from the sidebar

12 questions in this topic

Question 1 / 12

How does Python's reference counting work for memory management?

Reference counting is CPython's primary memory management mechanism. Every Python object has an internal reference count that tracks how many references point to it.

How it works:

Increment: When a new reference is created (assignment, function argument, adding to a container)
Decrement: When a reference is deleted (del), goes out of scope, or is reassigned
Deallocation: When the count reaches zero, the object is immediately freed

Advantages:

Deterministic: Objects are freed immediately when no longer referenced
Simple: Low overhead for the common case
Predictable: __del__ is called promptly

Limitations:

Cannot handle reference cycles (A references B, B references A)
Overhead of maintaining counts on every reference change
Not thread-safe without the GIL

Pseudo Code

Object lifecycle:
  x = SomeObject()   # refcount = 1
  y = x              # refcount = 2
  z = [x]            # refcount = 3
  del y              # refcount = 2
  z.pop()            # refcount = 1
  del x              # refcount = 0 -> FREED immediately

Code

import sys

# sys.getrefcount returns the reference count
# Note: passing to getrefcount itself creates a temporary reference (+1)

a = [1, 2, 3]
print(sys.getrefcount(a))  # 2 (a + getrefcount argument)

b = a  # new reference
print(sys.getrefcount(a))  # 3 (a + b + getrefcount argument)

c = [a, a, a]  # three more references
print(sys.getrefcount(a))  # 6 (a + b + 3 in list + getrefcount)

del b
print(sys.getrefcount(a))  # 5

c.clear()
print(sys.getrefcount(a))  # 2 (a + getrefcount)

# Demonstrating immediate deallocation
class Tracked:
    def __init__(self, name):
        self.name = name
        print(f"Created {self.name}")

    def __del__(self):
        print(f"Destroyed {self.name}")

print("--- Creating ---")
obj = Tracked("obj1")
print("--- Replacing ---")
obj = Tracked("obj2")  # obj1 is freed immediately (refcount -> 0)
print("--- Deleting ---")
del obj  # obj2 is freed immediately
print("--- Done ---")

# Reference cycle (refcount never reaches 0)
class Node:
    def __init__(self, name):
        self.name = name
        self.next = None

a = Node("A")
b = Node("B")
a.next = b  # A -> B
b.next = a  # B -> A (cycle!)

# Even after deleting a and b, the cycle keeps refcount > 0
# The cyclic garbage collector must handle this
del a, b  # objects not freed! (refcount = 1 each due to cycle)

2
3
6
5
2
--- Creating ---
Created obj1
--- Replacing ---
Created obj2
Destroyed obj1
--- Deleting ---
Destroyed obj2
--- Done ---

Key Points

•Every object has an internal reference count tracking pointers to it
•Objects are freed immediately when refcount reaches zero
•sys.getrefcount() returns the count (adds 1 for the argument itself)
•Reference counting cannot handle cyclic references -- needs cyclic GC

Gotchas

•sys.getrefcount() always shows one more than expected (the argument reference)
•Reference cycles prevent refcount from reaching zero -- leads to leaks without cyclic GC
•__del__ timing depends on reference counting -- not guaranteed in all Python implementations
•Interned objects (small ints, string literals) have artificially high refcounts

Every Python object has a reference count. When it reaches zero, the object is immediately freed. sys.getrefcount(obj) shows the count. Cycles (A->B->A) cannot be freed by refcounting alone -- the cyclic GC handles those.

•refcount=0 triggers immediate deallocation
•Cannot handle reference cycles

import sys

x = [1, 2, 3]
print(sys.getrefcount(x))  # 2 (x + argument to getrefcount)
y = x
print(sys.getrefcount(x))  # 3
del y
print(sys.getrefcount(x))  # 2

2
3
2

sys.getrefcount adds 1 because passing the object is itself a reference

Question 2 / 12

How does Python's cyclic garbage collector work?

The cyclic garbage collector supplements reference counting by detecting and collecting reference cycles -- groups of objects that reference each other but are not reachable from the program.

How it works (generational collection):

Python uses a generational approach with three generations (0, 1, 2):

Generation 0: Newly created objects. Collected most frequently.
Generation 1: Objects that survived one collection.
Generation 2: Long-lived objects. Collected least frequently.

Collection algorithm:

For each object in a generation, count internal references (references from other tracked objects)
Subtract internal references from the total refcount
Objects with effective refcount 0 after subtraction are unreachable (part of a cycle)
These objects are collected (freed)

Thresholds:

Collection triggers when allocations - deallocations > threshold for a generation. Default thresholds: (700, 10, 10).

Pseudo Code

Generational GC:
  Gen 0 (new objects): collected every ~700 allocations
  Gen 1 (survivors): collected every ~10 Gen 0 collections
  Gen 2 (long-lived): collected every ~10 Gen 1 collections

Cycle detection:
  For each object in generation:
    if reachable from root (program variables): keep
    if only reachable from other cycle members: collect

Code

import gc

# Check GC status
print(f"GC enabled: {gc.isenabled()}")  # True
print(f"Thresholds: {gc.get_threshold()}")  # (700, 10, 10)
print(f"Counts: {gc.get_count()}")  # (allocations_since_last_gc, gen1_count, gen2_count)

# Create a reference cycle
class Node:
    def __init__(self, name):
        self.name = name
        self.ref = None
    def __repr__(self):
        return f"Node({self.name})"

a = Node("A")
b = Node("B")
a.ref = b
b.ref = a  # cycle: A <-> B

# Delete external references
del a, b
# Objects still exist (refcount > 0 due to cycle)

# Force garbage collection
collected = gc.collect()  # returns number of unreachable objects found
print(f"Collected {collected} objects")  # 2 (A and B)

# Monitor GC activity
gc.set_debug(gc.DEBUG_STATS)  # print collection stats
gc.collect()
gc.set_debug(0)  # turn off

# Adjust thresholds (tune for your workload)
gc.set_threshold(1000, 15, 15)  # less frequent collections

# Get objects tracked by GC
class Tracked:
    pass

obj = Tracked()
tracked = gc.get_referrers(obj)  # objects that reference obj
print(f"Referrers: {len(tracked)}")

# Disable GC (for performance-critical sections)
gc.disable()
# ... performance-critical code ...
gc.enable()

# Generation info
for gen in range(3):
    objects = gc.get_objects(gen)
    print(f"Generation {gen}: {len(objects)} objects")

# Callbacks for GC events (Python 3.3+)
def gc_callback(phase, info):
    if phase == 'start':
        print(f"GC starting: generation {info['generation']}")
    elif phase == 'stop':
        print(f"GC done: collected {info['collected']}, uncollectable {info['uncollectable']}")

gc.callbacks.append(gc_callback)
gc.collect()
gc.callbacks.clear()

GC enabled: True
Thresholds: (700, 10, 10)
Counts: (13, 9, 0)
Collected 2 objects
Referrers: 1
Generation 0: 9 objects
Generation 1: 0 objects
Generation 2: 6645 objects
GC starting: generation 2
GC done: collected 0, uncollectable 0

Key Points

•Cyclic GC detects and frees reference cycles that refcounting misses
•Uses three generations: 0 (new), 1 (survived), 2 (long-lived)
•gc.collect() forces immediate collection; gc.get_threshold() shows trigger points
•gc.disable() can improve performance when you know there are no cycles

Gotchas

•Objects with __del__ in a cycle were uncollectable before Python 3.4 (PEP 442 fixed this)
•gc.disable() can cause memory leaks if your code creates reference cycles
•GC pauses can cause latency spikes -- tune thresholds for latency-sensitive applications
•Weak references (weakref) are a better solution than relying on GC for cycle-prone patterns

Python's cyclic GC detects and frees reference cycles using a generational algorithm (3 generations). It supplements reference counting. Use gc.collect() to force collection, gc.get_threshold() to see triggers.

•Three generations with increasing collection frequency
•Detects reference cycles that refcounting cannot handle

import gc

print(gc.get_threshold())  # (700, 10, 10)
print(gc.get_count())      # current allocation counts

collected = gc.collect()   # force collection
print(f"Collected {collected} cyclic objects")

(700, 10, 10)
(681, 8, 0)
Collected 6 cyclic objects

Disabling GC causes leaks if your code creates reference cycles

Question 3 / 12

What is del and why should you avoid relying on it?

__del__ is a finalizer method called when an object is about to be garbage collected. It is not a destructor in the C++ sense.

Problems with del:

Non-deterministic timing: With cyclic GC, you don't know when (or if) it will be called
Reference resurrection: Accessing self in __del__ can accidentally revive the object
Exception suppression: Exceptions in __del__ are ignored (printed to stderr)
GC complications: Before Python 3.4, objects with __del__ in cycles were uncollectable
Interpreter shutdown: During shutdown, __del__ may fail because globals are already cleaned up

Better alternatives:

Context managers (with statement + __enter__/__exit__)
weakref.finalize() for guaranteed cleanup
Explicit .close() methods

Pseudo Code

__del__ problems:
  1. When is it called? Unknown (depends on GC)
  2. Will it be called? Not guaranteed (interpreter shutdown)
  3. Is it safe? No (globals may be None)

Better: use context managers
  with Resource() as r:
      use(r)
  # __exit__ called deterministically

Code

import weakref
import gc

# __del__ basic usage (with caveats)
class Resource:
    def __init__(self, name):
        self.name = name
        print(f"Acquired {self.name}")

    def __del__(self):
        print(f"Released {self.name}")

# Deterministic: refcount drops to 0
r = Resource("A")
del r  # prints "Released A" immediately

# Non-deterministic: reference cycle
class LinkedNode:
    def __init__(self, name):
        self.name = name
        self.next = None
    def __del__(self):
        print(f"Finalizing {self.name}")

a = LinkedNode("A")
b = LinkedNode("B")
a.next = b
b.next = a
del a, b
# __del__ will be called eventually by cyclic GC, but timing is unknown
gc.collect()  # force it

# BETTER: Context manager
class ManagedResource:
    def __init__(self, name):
        self.name = name
        print(f"Acquired {self.name}")

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.close()
        return False

    def close(self):
        print(f"Released {self.name}")

with ManagedResource("B") as r:
    pass  # Released deterministically when exiting 'with'

# BETTER: weakref.finalize (guaranteed, no resurrection risk)
class SafeResource:
    def __init__(self, name):
        self.name = name
        self._finalizer = weakref.finalize(self, self._cleanup, name)

    @staticmethod
    def _cleanup(name):
        print(f"Finalized {name}")

    def close(self):
        self._finalizer()

obj = SafeResource("C")
del obj  # prints "Finalized C"

# weakref.finalize is called even during interpreter shutdown
# and handles reference cycles correctly

Acquired A
Released A
Finalizing A
Finalizing B
Acquired B
Released B
Finalized C

Key Points

•__del__ is a finalizer called when an object is about to be freed
•Timing is non-deterministic when cyclic GC is involved
•Context managers (__enter__/__exit__) are the preferred alternative
•weakref.finalize() provides guaranteed cleanup without __del__ pitfalls

Gotchas

•Exceptions in __del__ are silently ignored (only printed to stderr)
•__del__ can accidentally resurrect objects by creating new references to self
•During interpreter shutdown, module globals may already be None when __del__ runs
•__del__ is NOT guaranteed to be called for objects that still exist when the interpreter exits

__del__ is a finalizer called before garbage collection, but its timing is unpredictable. Use context managers (with statement) or weakref.finalize() instead for deterministic cleanup.

•__del__ timing is unpredictable
•Use context managers or weakref.finalize() instead

import weakref

# Avoid __del__, use weakref.finalize
class Resource:
    def __init__(self):
        weakref.finalize(self, print, "Cleaned up!")

r = Resource()
del r  # prints "Cleaned up!"

Exceptions in __del__ are silently suppressed

Question 4 / 12

How do you profile memory usage in Python?

Python provides several tools for memory profiling:

Built-in: `tracemalloc` (Python 3.4+)

Traces memory allocations back to their source file and line
Compare snapshots to find leaks
No external dependencies

Built-in: `sys.getsizeof()`

Returns the size of a single object in bytes
Does NOT include sizes of referenced objects

Third-party: `memory_profiler`

Line-by-line memory usage with @profile decorator
mprof command-line tool for plotting

Third-party: `objgraph`

Visualizes object reference graphs
Finds objects that are growing in count

Third-party: `pympler`

asizeof(): Deep size including referenced objects
Object tracking and statistics

Pseudo Code

tracemalloc workflow:
  1. tracemalloc.start()
  2. ... run code ...
  3. snapshot = tracemalloc.take_snapshot()
  4. stats = snapshot.statistics('lineno')
  5. Review top memory consumers

Leak detection:
  1. snapshot1 = take_snapshot()
  2. ... run suspected leaky code ...
  3. snapshot2 = take_snapshot()
  4. diff = snapshot2.compare_to(snapshot1, 'lineno')
  5. Review what grew

Code

import tracemalloc
import sys

# tracemalloc: trace memory allocations
tracemalloc.start()

# Allocate some memory
data = [list(range(1000)) for _ in range(100)]
small = {i: str(i) for i in range(1000)}

snapshot = tracemalloc.take_snapshot()
stats = snapshot.statistics('lineno')

print("Top 5 memory consumers:")
for stat in stats[:5]:
    print(f"  {stat}")

# Compare snapshots for leak detection
snapshot1 = tracemalloc.take_snapshot()

# Simulate memory growth
more_data = [bytearray(10000) for _ in range(50)]

snapshot2 = tracemalloc.take_snapshot()
diff = snapshot2.compare_to(snapshot1, 'lineno')

print("\nMemory changes:")
for stat in diff[:5]:
    print(f"  {stat}")

tracemalloc.stop()

# sys.getsizeof: size of individual objects (shallow)
print(f"\nint: {sys.getsizeof(0)} bytes")
print(f"str 'hello': {sys.getsizeof('hello')} bytes")
print(f"list [1,2,3]: {sys.getsizeof([1,2,3])} bytes")  # container only!
print(f"dict {{1:2}}: {sys.getsizeof({1: 2})} bytes")
print(f"empty dict: {sys.getsizeof({})} bytes")

# Deep size estimation
def deep_getsizeof(obj, seen=None):
    """Recursively calculate total size of object and its contents."""
    if seen is None:
        seen = set()
    obj_id = id(obj)
    if obj_id in seen:
        return 0
    seen.add(obj_id)
    size = sys.getsizeof(obj)
    if isinstance(obj, dict):
        size += sum(deep_getsizeof(k, seen) + deep_getsizeof(v, seen)
                    for k, v in obj.items())
    elif isinstance(obj, (list, tuple, set, frozenset)):
        size += sum(deep_getsizeof(item, seen) for item in obj)
    return size

data = {'key': [1, 2, 3], 'other': 'hello'}
print(f"\nShallow size: {sys.getsizeof(data)} bytes")
print(f"Deep size: {deep_getsizeof(data)} bytes")

# Current memory usage
print(f"\nCurrent: {tracemalloc.get_traced_memory() if tracemalloc.is_tracing() else 'not tracing'}")

Top 5 memory consumers:
<string>:8: size=3109 KiB, count=74501, average=43 B
<string>:9: size=102 KiB, count=1745, average=60 B

Memory changes:
<string>:22: size=492 KiB (+492 KiB), count=102 (+102), average=4934 B
/opt/homebrew/Cellar/python@3.12/3.12.13/Frameworks/Python.framework/Versions/3.12/lib/python3.12/tracemalloc.py:560: size=616 B (+304 B), count=4 (+2), average=154 B
/opt/homebrew/Cellar/python@3.12/3.12.13/Frameworks/Python.framework/Versions/3.12/lib/python3.12/tracemalloc.py:423: size=616 B (+304 B), count=4 (+2), average=154 B
/opt/homebrew/Cellar/python@3.12/3.12.13/Frameworks/Python.framework/Versions/3.12/lib/python3.12/tracemalloc.py:313: size=48 B (+48 B), count=1 (+1), average=48 B
<string>:8: size=3109 KiB (+0 B), count=74501 (+0), average=43 B

int: 28 bytes
str 'hello': 46 bytes
list [1,2,3]: 88 bytes
dict {1:2}: 224 bytes
empty dict: 64 bytes

Shallow size: 184 bytes
Deep size: 492 bytes

Current: not tracing

Key Points

•tracemalloc traces allocations to source lines -- best built-in tool
•Compare snapshots to find memory leaks
•sys.getsizeof() only measures shallow size (container itself, not contents)
•Use tracemalloc.start() at program start for best results

Gotchas

•sys.getsizeof() does NOT include the size of referenced objects (shallow only)
•tracemalloc itself uses memory -- adds ~10-30% overhead
•tracemalloc only tracks Python allocations, not C extension memory
•Memory profiler results can vary between runs due to GC timing

Use tracemalloc to trace memory allocations to source lines and compare snapshots for leak detection. sys.getsizeof() shows shallow size of individual objects.

•tracemalloc for allocation tracing and leak detection
•sys.getsizeof for shallow object size

import tracemalloc

tracemalloc.start()
data = [i ** 2 for i in range(10000)]
snapshot = tracemalloc.take_snapshot()
for stat in snapshot.statistics('lineno')[:3]:
    print(stat)
tracemalloc.stop()

<string>:4: size=395 KiB, count=9984, average=41 B

sys.getsizeof is shallow -- does not include referenced objects

Question 5 / 12

How does slots optimize memory usage in Python classes?

By default, Python objects store instance attributes in a __dict__ dictionary. __slots__ replaces this dict with a fixed set of slots, which are stored more efficiently.

Memory savings:

A __dict__ costs ~100+ bytes per instance (dict overhead)
__slots__ attributes use fixed offsets like C struct members
For objects with few attributes, savings can be 30-50% per instance
Significant when creating millions of instances

How to use:

class Point:
    __slots__ = ('x', 'y')

Trade-offs:

No __dict__ means you cannot add arbitrary attributes
No __weakref__ unless explicitly included in __slots__
Cannot use multiple inheritance with conflicting __slots__
Subclasses without __slots__ get __dict__ back

Pseudo Code

Without __slots__:
  instance.__dict__ = {'x': 1, 'y': 2}  # dict overhead per instance

With __slots__:
  instance.x = 1  # stored at fixed memory offset
  instance.y = 2  # stored at fixed memory offset
  # no __dict__ overhead

Code

import sys

# Without __slots__
class RegularPoint:
    def __init__(self, x, y):
        self.x = x
        self.y = y

# With __slots__
class SlottedPoint:
    __slots__ = ('x', 'y')

    def __init__(self, x, y):
        self.x = x
        self.y = y

# Memory comparison
reg = RegularPoint(1, 2)
slot = SlottedPoint(1, 2)

print(f"Regular: {sys.getsizeof(reg)} + dict {sys.getsizeof(reg.__dict__)} bytes")
print(f"Slotted: {sys.getsizeof(slot)} bytes (no __dict__)")

# Demonstrate: no arbitrary attributes with __slots__
try:
    slot.z = 3  # AttributeError!
except AttributeError as e:
    print(f"Cannot add attributes: {e}")

# Regular class allows arbitrary attributes
reg.z = 3  # works fine
print(f"Regular z: {reg.z}")

# Memory at scale
import tracemalloc
tracemalloc.start()

regular_points = [RegularPoint(i, i) for i in range(100_000)]
reg_size = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

tracemalloc.start()
slotted_points = [SlottedPoint(i, i) for i in range(100_000)]
slot_size = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

print(f"\n100K Regular objects: {reg_size / 1024 / 1024:.1f} MB")
print(f"100K Slotted objects: {slot_size / 1024 / 1024:.1f} MB")
print(f"Savings: {(1 - slot_size / reg_size) * 100:.0f}%")

# __slots__ with inheritance
class SlottedPoint3D(SlottedPoint):
    __slots__ = ('z',)  # add z slot, inherit x and y

    def __init__(self, x, y, z):
        super().__init__(x, y)
        self.z = z

p3d = SlottedPoint3D(1, 2, 3)
print(f"\n3D Point: ({p3d.x}, {p3d.y}, {p3d.z})")

# Including __weakref__ and __dict__ in slots
class FlexibleSlots:
    __slots__ = ('x', 'y', '__dict__', '__weakref__')

    def __init__(self, x, y):
        self.x = x
        self.y = y

f = FlexibleSlots(1, 2)
f.z = 3  # works because __dict__ is in __slots__
print(f"Flexible z: {f.z}")

Regular: 48 + dict 296 bytes
Slotted: 48 bytes (no __dict__)
Cannot add attributes: 'SlottedPoint' object has no attribute 'z'
Regular z: 3

100K Regular objects: 12.2 MB
100K Slotted objects: 8.4 MB
Savings: 31%

3D Point: (1, 2, 3)
Flexible z: 3

Key Points

•__slots__ replaces per-instance __dict__ with fixed-offset storage
•Saves 30-50% memory per instance, significant at scale
•Prevents adding arbitrary attributes (can be a feature for correctness)
•Include '__weakref__' in __slots__ if you need weak references to instances

Gotchas

•Subclass without __slots__ reintroduces __dict__ (negating savings)
•Cannot use __slots__ with multiple inheritance if both parents define overlapping slots
•Pickling slotted objects requires __getstate__/__setstate__ or protocol 2+
•__slots__ only saves memory on instances, not on the class itself

__slots__ replaces the per-instance __dict__ with fixed attribute slots, saving 30-50% memory per instance. Prevents adding arbitrary attributes.

•No __dict__ means fixed attributes, lower memory
•Worth it when creating millions of instances

class Point:
    __slots__ = ('x', 'y')
    def __init__(self, x, y):
        self.x = x
        self.y = y

p = Point(1, 2)
# p.z = 3  # AttributeError -- no __dict__

Subclass without __slots__ brings back __dict__

Question 6 / 12

What are weak references (weakref) and when should you use them?

A weak reference is a reference to an object that does not increase its reference count. If the only remaining references to an object are weak references, the object is garbage collected.

The `weakref` module provides:

weakref.ref(obj, callback=None) -- create a weak reference
weakref.proxy(obj) -- transparent proxy that raises ReferenceError when dead
weakref.WeakValueDictionary -- dict where values are weak references
weakref.WeakKeyDictionary -- dict where keys are weak references
weakref.WeakSet -- set with weak references
weakref.finalize(obj, func, *args) -- call func when obj is collected

Use cases:

Caches: Store values that can be reclaimed when memory is low
Observer pattern: Avoid preventing observed objects from being collected
Breaking reference cycles: Replace one direction with a weak reference
Object tracking: Monitor objects without extending their lifetime

Pseudo Code

Strong reference: obj has refcount >= 1, kept alive
Weak reference: does not count, obj can be collected

ref = weakref.ref(obj)  # weak reference
obj_again = ref()       # call to dereference
if obj_again is not None:
    use(obj_again)
# If obj was collected, ref() returns None

Code

import weakref

class ExpensiveObject:
    def __init__(self, name):
        self.name = name
    def __repr__(self):
        return f"ExpensiveObject({self.name!r})"

# Basic weak reference
obj = ExpensiveObject("data")
weak = weakref.ref(obj)

print(weak())       # ExpensiveObject('data') -- object still alive
print(weak() is obj)  # True

del obj
print(weak())  # None -- object was collected

# Weak reference with callback
def on_finalize(ref):
    print(f"Object was collected! ref() = {ref()}")

obj = ExpensiveObject("tracked")
weak = weakref.ref(obj, on_finalize)
del obj  # prints callback message

# WeakValueDictionary: cache that allows GC
cache = weakref.WeakValueDictionary()

def get_or_create(key):
    if key in cache:
        return cache[key]
    obj = ExpensiveObject(key)
    cache[key] = obj
    return obj

result = get_or_create("item1")
print(f"Cache size: {len(cache)}")  # 1
print(cache["item1"])  # works, result keeps it alive

del result  # no more strong references
import gc; gc.collect()
print(f"Cache size after del: {len(cache)}")  # 0 (collected)

# WeakSet: tracking instances without preventing collection
class Plugin:
    _instances = weakref.WeakSet()

    def __init__(self, name):
        self.name = name
        Plugin._instances.add(self)

    @classmethod
    def get_active(cls):
        return list(cls._instances)

p1 = Plugin("auth")
p2 = Plugin("logging")
print(f"Active plugins: {len(Plugin.get_active())}")  # 2

del p1
import gc; gc.collect()
print(f"After del p1: {len(Plugin.get_active())}")  # 1

# weakref.finalize: safe destructor alternative
obj = ExpensiveObject("resource")
finalizer = weakref.finalize(obj, print, "Resource cleaned up!")
print(f"Finalizer alive: {finalizer.alive}")  # True
del obj  # prints "Resource cleaned up!"

ExpensiveObject('data')
True
None
Object was collected! ref() = None
Cache size: 1
ExpensiveObject('item1')
Cache size after del: 0
Active plugins: 2
After del p1: 1
Finalizer alive: True
Resource cleaned up!

Key Points

•Weak references do not prevent garbage collection of the referenced object
•Call the weak ref to get the object (returns None if collected)
•WeakValueDictionary/WeakSet for caches and instance tracking
•weakref.finalize() is a better alternative to __del__

Gotchas

•Not all objects support weak references -- int, str, tuple, list, dict do not
•Weak references to objects with __slots__ require '__weakref__' in __slots__
•Accessing a dead weakref.proxy() raises ReferenceError (not None)
•WeakValueDictionary may not shrink immediately -- GC timing affects it

Weak references (weakref.ref) point to objects without preventing garbage collection. If the object is collected, the ref returns None. Use WeakValueDictionary for caches and WeakSet for instance tracking.

•Does not increase reference count
•Returns None when object is collected

import weakref

class Data:
    pass

obj = Data()
weak = weakref.ref(obj)
print(weak())  # <Data object>
del obj
print(weak())  # None -- collected

<__main__.Data object at <address>>
None

Built-in types (int, str, list) do not support weak references

Question 7 / 12

What is object interning and small integer caching in Python?

Python interns (caches and reuses) certain immutable objects to save memory and improve performance.

Small Integer Caching

CPython pre-creates integer objects for -5 to 256 at startup. Any integer in this range reuses the same object. Outside this range, new objects may be created.

String Interning

CPython automatically interns:

String literals that look like identifiers (alphanumeric + underscores)
Strings used as variable names, attribute names, dict keys
You can manually intern with sys.intern(string)

Why it matters:

is comparison is O(1) on interned objects (pointer comparison)
Saves memory for repeated identical values
But relying on interning for correctness is a bug (use == for value comparison)

Pseudo Code

Small integers:
  a = 100; b = 100
  a is b  -> True  (same cached object)

  a = 1000; b = 1000
  a is b  -> False (or True, implementation-dependent)

String interning:
  a = 'hello'; b = 'hello'
  a is b  -> True  (interned identifier-like string)

  a = 'hello world'; b = 'hello world'
  a is b  -> True or False (depends on context)

Code

import sys

# Small integer caching: -5 to 256
a = 256
b = 256
print(f"256: a is b = {a is b}")  # True (cached)
print(f"id(a)={id(a)}, id(b)={id(b)}")  # same id

a = 257
b = 257
print(f"257: a is b = {a is b}")  # May be False (outside cache range)

a = -5
b = -5
print(f"-5: a is b = {a is b}")   # True (cached)

a = -6
b = -6
print(f"-6: a is b = {a is b}")   # May be False

# String interning
a = 'hello'
b = 'hello'
print(f"'hello': a is b = {a is b}")  # True (identifier-like, interned)

a = 'hello world'
b = 'hello world'
print(f"'hello world': a is b = {a is b}")  # May be False (has space)

# Manual string interning
a = sys.intern('hello world')
b = sys.intern('hello world')
print(f"interned 'hello world': a is b = {a is b}")  # True

# Practical: memory savings with sys.intern
names = [sys.intern(f"user_{i % 100}") for i in range(10000)]
# Only 100 unique string objects instead of 10000

# None, True, False are singletons (always interned)
a = None
b = None
print(f"None: a is b = {a is b}")  # Always True
print(f"True: True is True = {True is True}")  # Always True

# Empty tuple and empty frozenset are also cached
a = ()
b = ()
print(f"Empty tuple: a is b = {a is b}")  # True

# IMPORTANT: Never use 'is' for value comparison!
x = 1000
y = 1000
# x is y may be True or False depending on context!
# ALWAYS use == for value comparison
print(f"1000 == 1000: {x == y}")  # Always True

256: a is b = True
id(a)=4387749536, id(b)=4387749536
257: a is b = True
-5: a is b = True
-6: a is b = True
'hello': a is b = True
'hello world': a is b = True
interned 'hello world': a is b = True
None: a is b = True
True: True is True = True
Empty tuple: a is b = True
1000 == 1000: True

Key Points

•CPython caches integers -5 to 256 as singletons
•Identifier-like strings are automatically interned
•sys.intern() manually interns strings for memory savings
•None, True, False, empty tuple are always singletons

Gotchas

•NEVER use 'is' for value comparison -- it tests identity, not equality
•Integer caching range (-5 to 256) is a CPython implementation detail, not guaranteed
•String interning behavior varies between REPL, scripts, and different Python versions
•In compiled .py files, the compiler may intern more values than in the REPL

CPython caches integers -5 to 256 and identifier-like strings as singletons (same object reused). Use sys.intern() to manually intern strings. Always use == for value comparison, never is.

•Integers -5 to 256 are pre-cached singletons
•Use == for values, 'is' only for None/True/False

import sys

a = 100; b = 100
print(a is b)  # True (cached integer)

a = 'hello'; b = 'hello'
print(a is b)  # True (interned string)

# Manual interning
a = sys.intern('hello world')
b = sys.intern('hello world')
print(a is b)  # True

True
True
True

Interning behavior is a CPython detail -- do not rely on it for correctness

Question 8 / 12

What are the common causes of memory leaks in Python?

While Python has garbage collection, memory leaks still occur. A memory leak in Python means objects that are no longer needed are still being referenced, preventing collection.

Common causes:

Reference cycles with __del__ (pre-3.4): Objects with finalizers in cycles were uncollectable
Global variables and module-level caches: Growing unbounded
Forgotten event listeners/callbacks: References kept in callback registries
Mutable default arguments: The default object persists across calls
Large closures: Closures capture entire enclosing scope
C extension leaks: Memory allocated in C not freed properly
Thread-local storage not cleaned up: Thread pool reuses threads
Caches without eviction: Growing forever (use LRU cache instead)
Logging handlers: Custom handlers holding references
Exception tracebacks: sys.exc_info() holds references to stack frames

Pseudo Code

Common patterns:
  1. cache = {}  # grows forever
     Fix: use LRU cache or WeakValueDictionary

  2. def register(callback):
         callbacks.append(callback)  # never removed
     Fix: use weak references

  3. def func(items=[]):  # mutable default
         items.append(1)  # accumulates!
     Fix: def func(items=None): items = items or []

Code

import gc
import weakref
import tracemalloc
from functools import lru_cache

# Leak 1: Unbounded cache
class LeakyCache:
    def __init__(self):
        self._cache = {}  # grows forever!

    def get(self, key):
        if key not in self._cache:
            self._cache[key] = self._expensive_compute(key)
        return self._cache[key]

    def _expensive_compute(self, key):
        return [0] * 10000  # big object

# Fix: use LRU cache or WeakValueDictionary
class FixedCache:
    def __init__(self, maxsize=100):
        self._cache = {}
        self._maxsize = maxsize

    def get(self, key):
        if key not in self._cache:
            if len(self._cache) >= self._maxsize:
                oldest = next(iter(self._cache))
                del self._cache[oldest]
            self._cache[key] = self._expensive_compute(key)
        return self._cache[key]

    def _expensive_compute(self, key):
        return [0] * 10000

# Leak 2: Mutable default argument
def leaky_function(items=[]):
    items.append("data")
    return items

leaky_function()  # ['data']
leaky_function()  # ['data', 'data'] -- accumulating!
leaky_function()  # ['data', 'data', 'data']

# Fix:
def fixed_function(items=None):
    if items is None:
        items = []
    items.append("data")
    return items

# Leak 3: Closure capturing large scope
def create_processor():
    large_data = [0] * 1_000_000  # captured by closure!

    def process(x):
        return x * 2  # doesn't even use large_data

    return process  # large_data stays alive

# Fix: only capture what you need
def create_processor_fixed():
    large_data = [0] * 1_000_000
    result = sum(large_data)  # extract what you need
    del large_data  # explicitly free

    def process(x):
        return x * result

    return process

# Leak 4: Exception traceback holding references
def leaky_exception_handling():
    try:
        raise ValueError("error")
    except ValueError:
        import sys
        exc_info = sys.exc_info()  # holds reference to stack frame!
        # Fix: del exc_info when done, or use 'except ValueError as e'

# Detecting leaks with tracemalloc
def find_leaks():
    tracemalloc.start()
    snapshot1 = tracemalloc.take_snapshot()

    # Run suspected leaky code
    cache = LeakyCache()
    for i in range(1000):
        cache.get(f"key_{i}")

    snapshot2 = tracemalloc.take_snapshot()
    diff = snapshot2.compare_to(snapshot1, 'lineno')

    print("Memory growth:")
    for stat in diff[:5]:
        print(f"  {stat}")

    tracemalloc.stop()

find_leaks()

# Using gc to find unreachable cycles
gc.collect()
print(f"Unreachable objects: {gc.garbage}")  # objects that couldn't be freed

Memory growth:
<string>:17: size=76.3 MiB (+76.3 MiB), count=1985 (+1985), average=39.4 KiB
<string>:89: size=46.8 KiB (+46.8 KiB), count=1000 (+1000), average=48 B
<string>:13: size=25.4 KiB (+25.4 KiB), count=1 (+1), average=25.4 KiB
<string>:87: size=312 B (+312 B), count=2 (+2), average=156 B
/opt/homebrew/Cellar/python@3.12/3.12.13/Frameworks/Python.framework/Versions/3.12/lib/python3.12/tracemalloc.py:560: size=312 B (+312 B), count=2 (+2), average=156 B
Unreachable objects: []

Key Points

•Unbounded caches are the most common cause -- use LRU or weak references
•Mutable default arguments accumulate data across calls
•Closures capture the entire enclosing scope, not just used variables
•tracemalloc snapshot comparison is the best tool for finding leaks

Gotchas

•gc.collect() may not free all memory -- C extensions may leak outside Python's control
•Even after fixing a leak, memory may not be returned to the OS (Python's memory allocator keeps pools)
•Thread pools reuse threads, so thread-local data persists between tasks
•Exception handlers with 'as e' hold references to tracebacks until the handler scope exits

Common Python memory leaks: unbounded caches (fix: LRU/weak refs), mutable default args (fix: None default), large closures (fix: capture only needed vars), forgotten callbacks. Use tracemalloc snapshots to detect leaks.

•Unbounded caches and mutable defaults are top causes
•Use tracemalloc to compare snapshots

# Leak: mutable default
def bad(items=[]): items.append(1); return items
bad()  # [1]
bad()  # [1, 1] -- growing!

# Fix:
def good(items=None):
    items = items or []
    items.append(1)
    return items

Memory freed by Python may not be returned to the OS

Question 9 / 12

How does sys.getrefcount() work and why is it always higher than expected?

sys.getrefcount(object) returns the reference count of an object. However, the returned value is always at least 1 higher than you might expect because the function call itself creates a temporary reference to the object (passed as an argument).

Why the count seems high:

The argument to getrefcount() is itself a reference (+1)
The variable holding the object (+1)
Any other references (containers, other variables)
CPython internal references (interned values, optimization caches)

When it's useful:

Debugging memory issues
Understanding object lifecycle
Verifying that references are properly cleaned up

Pseudo Code

x = SomeObject()
sys.getrefcount(x)
  -> x itself = 1 reference
  -> argument to getrefcount = 1 reference
  -> total = 2 (minimum for any named variable)

Code

import sys

# Basic refcount
a = object()
print(sys.getrefcount(a))  # 2 (a + getrefcount arg)

# Adding references
b = a
print(sys.getrefcount(a))  # 3 (a + b + getrefcount arg)

c = [a, a, a]
print(sys.getrefcount(a))  # 6 (a + b + 3 in list + getrefcount arg)

del b
print(sys.getrefcount(a))  # 5 (a + 3 in list + getrefcount arg)

c.clear()
print(sys.getrefcount(a))  # 2 (a + getrefcount arg)

# Small integers have very high refcounts (cached/interned)
print(sys.getrefcount(0))    # Very high (used internally)
print(sys.getrefcount(1))    # Very high
print(sys.getrefcount(None)) # Very high (singleton)

# Strings that look like identifiers are interned
print(sys.getrefcount('hello'))  # Higher than expected (interned)

# Creating temporary references
def show_refs(obj):
    # obj parameter is another reference!
    print(f"Inside function: {sys.getrefcount(obj)}")
    # getrefcount arg + obj parameter + original variable = 3+

my_obj = object()
print(f"Before call: {sys.getrefcount(my_obj)}")  # 2
show_refs(my_obj)  # 4 (my_obj + obj parameter + getrefcount arg + ???)
print(f"After call: {sys.getrefcount(my_obj)}")   # 2 (back to normal)

# Verifying cleanup
def verify_cleanup():
    data = [1, 2, 3]
    ref_before = sys.getrefcount(data)

    container = {'key': data}
    ref_during = sys.getrefcount(data)

    del container
    ref_after = sys.getrefcount(data)

    print(f"Before: {ref_before}, During: {ref_during}, After: {ref_after}")
    # Before: 2, During: 3, After: 2

verify_cleanup()

2
3
6
5
2
4294967295
4294967295
4294967295
3
Before call: 2
Inside function: 3
After call: 2
Before: 2, During: 3, After: 2

Key Points

•Always returns at least 2 for a named variable (variable + function argument)
•Useful for debugging reference leaks and understanding object lifecycle
•Small integers, None, and interned strings have very high counts
•The count decreases predictably as references are removed

Gotchas

•The +1 from getrefcount itself makes exact comparison tricky
•CPython's internal optimizations may hold additional references (bytecode cache, etc.)
•The REPL may hold extra references to recent results (_)
•getrefcount is CPython-specific -- other implementations may not have it

sys.getrefcount(obj) returns the reference count, always at least +1 higher than expected because passing the object to the function creates a temporary reference.

•Always +1 because the argument itself is a reference
•Small ints and None have very high counts (interned)

import sys

x = [1, 2, 3]
print(sys.getrefcount(x))  # 2 (x + getrefcount's argument)
y = x
print(sys.getrefcount(x))  # 3 (x + y + argument)
del y
print(sys.getrefcount(x))  # 2

2
3
2

CPython-specific; other implementations may not support it

Question 10 / 12

How does Python's memory allocator work internally?

CPython uses a hierarchical memory allocation system with three levels:

Level 1: OS Allocator (malloc/mmap)

Handles allocations > 512 bytes
Memory returned to the OS when freed

Level 2: Python's Object Allocator (pymalloc)

Handles allocations <= 512 bytes (most Python objects)
Uses arenas (256 KB chunks from the OS), divided into pools (4 KB), divided into blocks (8-512 bytes in 8-byte increments)
Pools are grouped by block size class

Level 3: Object-specific allocators

int, float, tuple, etc. have specialized free lists
Recently freed objects are kept in free lists for quick reallocation

Key insight:

Python often does NOT return memory to the OS even after objects are freed. pymalloc keeps arena memory for future allocations. Memory is only returned when an entire arena is empty.

Pseudo Code

Memory hierarchy:
  Arena (256 KB) -> Pool (4 KB) -> Block (8-512 bytes)

Allocation:
  if size <= 512 bytes: pymalloc (fast, pooled)
  if size > 512 bytes: malloc (OS allocator)

Deallocation:
  Block freed -> returned to pool's free list
  Pool empty -> returned to arena's free list
  Arena empty -> returned to OS

Code

import sys
import tracemalloc

# Object sizes (shows Python's memory overhead)
print("Object sizes:")
print(f"  int(0):        {sys.getsizeof(0)} bytes")
print(f"  int(2**30):    {sys.getsizeof(2**30)} bytes")
print(f"  float(0.0):    {sys.getsizeof(0.0)} bytes")
print(f"  str(''):       {sys.getsizeof('')} bytes")
print(f"  str('hello'):  {sys.getsizeof('hello')} bytes")
print(f"  list([]):      {sys.getsizeof([])} bytes")
print(f"  list([1]*10):  {sys.getsizeof([1]*10)} bytes")
print(f"  dict({{}}):      {sys.getsizeof({})} bytes")
print(f"  tuple(()):     {sys.getsizeof(())} bytes")
print(f"  set():         {sys.getsizeof(set())} bytes")

# Demonstrating memory not returned to OS
import os

def get_process_memory():
    """Get current process memory in MB (platform-dependent)."""
    try:
        import resource
        return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024  # macOS: bytes
    except ImportError:
        return 0

# Before allocation
mem_before = get_process_memory()

# Allocate lots of small objects
data = [bytearray(100) for _ in range(100_000)]
mem_during = get_process_memory()

# Free everything
del data
import gc
gc.collect()
mem_after = get_process_memory()

print(f"\nProcess memory:")
print(f"  Before: {mem_before:.1f} MB")
print(f"  During: {mem_during:.1f} MB")
print(f"  After:  {mem_after:.1f} MB")
print("  Note: 'After' may still be high -- Python keeps arena pools")

# Free lists for common types
# Python reuses recently freed objects of common types
# Visible with id() -- same id reused
for _ in range(3):
    x = [1, 2, 3]
    addr = id(x)
    del x
    y = [4, 5, 6]
    print(f"  Reused address: {id(y) == addr}")  # Often True

Object sizes:
int(0):        28 bytes
int(2**30):    32 bytes
float(0.0):    24 bytes
str(''):       41 bytes
str('hello'):  46 bytes
list([]):      56 bytes
list([1]*10):  136 bytes
dict({}):      64 bytes
tuple(()):     40 bytes
set():         216 bytes

Process memory:
Before: 14688.0 MB
During: 32752.0 MB
After:  32752.0 MB
Note: 'After' may still be high -- Python keeps arena pools
Reused address: True
Reused address: True
Reused address: True

Key Points

•pymalloc handles objects <= 512 bytes using arenas (256KB) -> pools (4KB) -> blocks
•Larger allocations go directly to the OS malloc
•Python often does NOT return freed memory to the OS (arenas kept for reuse)
•Free lists allow fast reallocation of recently freed common objects

Gotchas

•Process RSS (resident memory) may stay high even after freeing Python objects
•Memory fragmentation: if one object in an arena is alive, the whole arena is kept
•sys.getsizeof() shows the Python overhead but not the actual pymalloc block size
•Disabling pymalloc (PYTHONMALLOC=malloc) makes memory tools like Valgrind work better

CPython's pymalloc handles small objects (<= 512 bytes) using arenas (256KB) -> pools (4KB) -> blocks. Large objects use OS malloc. Freed memory is often NOT returned to the OS -- kept in pools for reuse.

•Small objects pooled in arenas; large objects use OS malloc
•Freed memory often stays with the process

import sys

# Python object overhead
print(sys.getsizeof(42))      # 28 bytes for an int!
print(sys.getsizeof('hello')) # 54 bytes for 5-char string
print(sys.getsizeof([]))      # 56 bytes for empty list

28
46
56

Process memory may not shrink after freeing objects

Question 11 / 12

How can you reduce memory usage when working with large datasets in Python?

Strategies for reducing memory usage:

Use generators instead of lists: Process items one at a time instead of loading all into memory
__slots__: Save 30-50% per instance for data classes
NumPy arrays: Much more compact than Python lists for numeric data
Tuples instead of lists: Slightly smaller footprint
Interning strings: sys.intern() for repeated string values
Memory-mapped files: Process files without loading entirely
Data type optimization: Use appropriate types (int vs float, array module)
Streaming/chunked processing: Process data in chunks
del and gc.collect(): Explicitly free large objects
Compress in-memory data: zlib/lz4 for data you access infrequently

Pseudo Code

Memory hierarchy (most to least efficient for numbers):
  numpy.int32 array:  4 bytes per element
  array.array('i'):   4 bytes per element
  tuple of ints:      ~28 bytes per element (object overhead)
  list of ints:       ~28 bytes per element + list overhead

Code

import sys
from array import array
import mmap

# 1. Generators vs Lists
def sum_with_list(n):
    return sum([i * i for i in range(n)])  # stores entire list in memory

def sum_with_generator(n):
    return sum(i * i for i in range(n))  # processes one at a time

# 2. Compare memory of different containers
n = 10000
lst = list(range(n))
tpl = tuple(range(n))
arr = array('i', range(n))  # C-type int array

print(f"list: {sys.getsizeof(lst):,} bytes")
print(f"tuple: {sys.getsizeof(tpl):,} bytes")
print(f"array: {sys.getsizeof(arr):,} bytes")

# 3. __slots__ for data objects
class RegularRecord:
    def __init__(self, name, value):
        self.name = name
        self.value = value

class SlottedRecord:
    __slots__ = ('name', 'value')
    def __init__(self, name, value):
        self.name = name
        self.value = value

import tracemalloc
tracemalloc.start()
reg = [RegularRecord(f"item_{i}", i) for i in range(10000)]
reg_mem = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

tracemalloc.start()
slot = [SlottedRecord(f"item_{i}", i) for i in range(10000)]
slot_mem = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

print(f"\nRegular: {reg_mem / 1024:.0f} KB")
print(f"Slotted: {slot_mem / 1024:.0f} KB")
print(f"Savings: {(1 - slot_mem / reg_mem) * 100:.0f}%")

# 4. String interning for repeated values
categories = ['electronics', 'clothing', 'food'] * 10000

tracemalloc.start()
normal = [cat for cat in categories]
normal_mem = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

tracemalloc.start()
interned = [sys.intern(cat) for cat in categories]
interned_mem = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

print(f"\nNormal strings: {normal_mem / 1024:.0f} KB")
print(f"Interned strings: {interned_mem / 1024:.0f} KB")

# 5. Chunked file processing
def process_large_file(filepath, chunk_size=8192):
    """Process a file in chunks without loading it all."""
    with open(filepath, 'rb') as f:
        while chunk := f.read(chunk_size):
            # process chunk
            pass

# 6. Memory-mapped file
def search_large_file(filepath, pattern):
    """Search a large file using memory mapping."""
    with open(filepath, 'rb') as f:
        with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
            index = mm.find(pattern.encode())
            return index

# 7. Using __sizeof__ for custom objects
class EfficientBuffer:
    __slots__ = ('_data',)

    def __init__(self, size):
        self._data = bytearray(size)

    def __sizeof__(self):
        return super().__sizeof__() + sys.getsizeof(self._data)

buf = EfficientBuffer(1000)
print(f"\nBuffer size: {sys.getsizeof(buf)} bytes")

list: 80,056 bytes
tuple: 80,040 bytes
array: 40,420 bytes

Regular: 1659 KB
Slotted: 1344 KB
Savings: 19%

Normal strings: 241 KB
Interned strings: 241 KB

Buffer size: 1097 bytes

Key Points

•Generators process items lazily -- O(1) memory instead of O(n)
•__slots__ saves 30-50% per instance for data objects
•array.array and numpy are far more compact than lists for numeric data
•sys.intern() deduplicates repeated strings

Gotchas

•Generators can only be iterated once -- convert to list if you need multiple passes
•memory-mapped files on 32-bit systems are limited to ~2GB
•Premature optimization: profile first, optimize the actual bottleneck
•Some memory-saving techniques (compression, generators) trade CPU time for memory

Key strategies: generators (lazy), __slots__ (data objects), array/numpy (numeric data), sys.intern() (repeated strings), chunked processing (large files), memory-mapped files.

•Generators, __slots__, array/numpy for memory savings
•Profile first, optimize bottlenecks

# Generator: O(1) memory
total = sum(x*x for x in range(1_000_000))  # no list in memory

# array: compact numeric storage
from array import array
arr = array('i', range(10000))  # ~40KB vs ~280KB for list

# __slots__: smaller objects
class Point:
    __slots__ = ('x', 'y')
    def __init__(self, x, y): self.x = x; self.y = y

Generators are single-use -- cannot iterate twice

Question 12 / 12

What is the difference between shallow copy and deep copy in terms of memory?

Shallow Copy

Creates a new container object but shares references to the same child objects. Only the top-level structure is copied.

Deep Copy

Recursively copies all objects, creating completely independent duplicates. No shared references.

Memory implications:

Shallow copy: Low memory cost (only copies the container, shares contents)
Deep copy: High memory cost (duplicates everything recursively)

Methods:

copy.copy() -- shallow copy
copy.deepcopy() -- deep copy
list(), dict(), .copy() -- shallow copy
Slicing a[:] -- shallow copy for lists

Pseudo Code

Original: [obj_A, obj_B, obj_C]

Shallow copy: new_list -> [ref_to_A, ref_to_B, ref_to_C]
  Modifying obj_A affects both original and copy
  New list container, same objects inside

Deep copy: new_list -> [copy_of_A, copy_of_B, copy_of_C]
  Modifying copy_of_A does NOT affect original
  Everything is independent

Code

import copy
import sys

# Shallow copy: new container, shared contents
original = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
shallow = copy.copy(original)

print(f"Same outer list? {original is shallow}")            # False
print(f"Same inner list? {original[0] is shallow[0]}")      # True (shared!)

# Modifying shared inner object affects both
shallow[0].append(99)
print(f"Original: {original[0]}")  # [1, 2, 3, 99] -- affected!
print(f"Shallow:  {shallow[0]}")   # [1, 2, 3, 99]

# But modifying the outer container is independent
shallow.append([10, 11, 12])
print(f"Original length: {len(original)}")  # 3
print(f"Shallow length:  {len(shallow)}")   # 4

# Deep copy: everything independent
original = [[1, 2, 3], [4, 5, 6]]
deep = copy.deepcopy(original)

print(f"\nSame outer? {original is deep}")          # False
print(f"Same inner? {original[0] is deep[0]}")      # False (independent!)

deep[0].append(99)
print(f"Original: {original[0]}")  # [1, 2, 3] -- unaffected!
print(f"Deep:     {deep[0]}")      # [1, 2, 3, 99]

# Memory comparison
import tracemalloc

big_list = [[i] * 100 for i in range(1000)]

tracemalloc.start()
shallow_copy = copy.copy(big_list)
shallow_mem = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

tracemalloc.start()
deep_copy = copy.deepcopy(big_list)
deep_mem = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

print(f"\nShallow copy memory: {shallow_mem:,} bytes")
print(f"Deep copy memory:    {deep_mem:,} bytes")
print(f"Deep/Shallow ratio:  {deep_mem / shallow_mem:.1f}x")

# Deep copy handles cycles correctly
a = [1, 2]
b = [3, 4]
a.append(b)
b.append(a)  # cycle: a -> b -> a

deep_a = copy.deepcopy(a)  # handles cycle, no infinite recursion
print(f"\nCycle handled: {deep_a[2][2] is deep_a}")  # True (internal cycle preserved)
print(f"Independent: {deep_a is not a}")  # True

# Custom copy behavior with __copy__ and __deepcopy__
class Config:
    def __init__(self, settings, cache=None):
        self.settings = settings
        self.cache = cache or {}

    def __copy__(self):
        # Shallow copy settings, reset cache
        return Config(self.settings.copy(), cache={})

    def __deepcopy__(self, memo):
        # Deep copy settings, reset cache
        return Config(copy.deepcopy(self.settings, memo), cache={})

config = Config({'debug': True, 'db': {'host': 'localhost'}})
config_copy = copy.deepcopy(config)
print(f"Independent settings: {config.settings is not config_copy.settings}")  # True
print(f"Cache reset: {config_copy.cache}")  # {}

Same outer list? False
Same inner list? True
Original: [1, 2, 3, 99]
Shallow:  [1, 2, 3, 99]
Original length: 3
Shallow length:  4

Same outer? False
Same inner? False
Original: [1, 2, 3]
Deep:     [1, 2, 3, 99]

Shallow copy memory: 8,000 bytes
Deep copy memory:    928,912 bytes
Deep/Shallow ratio:  116.1x

Cycle handled: True
Independent: True
Independent settings: True
Cache reset: {}

Key Points

•Shallow copy: new container, shared child objects (low memory cost)
•Deep copy: recursive duplication of all objects (high memory cost)
•deepcopy handles reference cycles correctly (uses memo dict)
•Customize with __copy__ and __deepcopy__ methods

Gotchas

•Shallow copying a list of mutable objects means changes to inner objects affect both copies
•deepcopy can be very slow for large nested structures with many shared references
•deepcopy of objects with open file handles or network connections can break
•For flat lists of immutables (ints, strings), shallow copy is effectively the same as deep copy

Shallow copy (copy.copy()) creates a new container but shares inner objects. Deep copy (copy.deepcopy()) recursively duplicates everything. Shallow is fast/low-memory, deep is safe/high-memory.

•Shallow: new container, shared contents
•Deep: everything duplicated recursively

import copy

original = [[1, 2], [3, 4]]
shallow = copy.copy(original)
deep = copy.deepcopy(original)

shallow[0].append(99)
print(original[0])  # [1, 2, 99] -- shared!

deep[1].append(99)
print(original[1])  # [3, 4] -- independent!

[1, 2, 99]
[3, 4]

Shallow copy of mutable objects shares inner state -- changes propagate

Built by onllm.dev

Home GitHub Support awesomePrep

How does Python's reference counting work for memory management?

How it works:

Advantages:

Limitations:

How does Python's cyclic garbage collector work?

How it works (generational collection):

Collection algorithm:

Thresholds:

What is __del__ and why should you avoid relying on it?

Problems with del:

Better alternatives:

How do you profile memory usage in Python?

Built-in: tracemalloc (Python 3.4+)

Built-in: sys.getsizeof()

Third-party: memory_profiler

Third-party: objgraph

Third-party: pympler

How does __slots__ optimize memory usage in Python classes?

Memory savings:

How to use:

Trade-offs:

What are weak references (weakref) and when should you use them?

The weakref module provides:

Use cases:

What is object interning and small integer caching in Python?

Small Integer Caching

String Interning

Why it matters:

What are the common causes of memory leaks in Python?

Common causes:

How does sys.getrefcount() work and why is it always higher than expected?

Why the count seems high:

When it's useful:

How does Python's memory allocator work internally?

Level 1: OS Allocator (malloc/mmap)

Level 2: Python's Object Allocator (pymalloc)

Level 3: Object-specific allocators

Key insight:

How can you reduce memory usage when working with large datasets in Python?

Strategies for reducing memory usage:

What is the difference between shallow copy and deep copy in terms of memory?

Shallow Copy

Deep Copy

Memory implications:

Methods:

Memory Management & Garbage Collection

What is del and why should you avoid relying on it?

Built-in: `tracemalloc` (Python 3.4+)

Built-in: `sys.getsizeof()`

Third-party: `memory_profiler`

Third-party: `objgraph`

Third-party: `pympler`

How does slots optimize memory usage in Python classes?

The `weakref` module provides: