I’ll give some background to what is occurring, provide a few workarounds. The codepoint ud83d is a High Surrogate, and does not exist as a Unicode character in and of itself. Plane 0 (the Basic Multiplingual Plane) contains the Unicode codepoints 0x0000–0xD7FF and 0xE000–0xFFFF. The characters 0xD800–0xDFFF are reserved for surrogate pairs and should not occur in Python string objects.

The encodings UTF-16, MUTF-8 and CESU-8 encode characters from 0x10000 onwards as a pair of codepoints (refered to as surrogate pairs). The made up of a high surrogate (0xD800–0xDBFF) and a low surrogate (0xDC00–0xDFFF).

If we take the cat emoji (U+1F408) as an example:

s = '🐈'
print('UTF-8: ', s.encode('utf-8').hex(' ').upper())
# UTF-8:  F0 9F 90 88
print('UTF-16-BE: ', s.encode('utf-16-be').hex(' ').upper())
# UTF-16-BE:  D8 3D DC 08

In UTF-8 it is represented as a sequence of four bytes representing a single Unicode codepoint, U+1F408. In UTF-16 it is a sequence of four bytes representing a surrogate pair U+D83D U+DC08.

There are two scenarios likely in your data:

1. U+D83D is the first character in a surrogate pair, or
2. U+D83D is a lone surrogate (with no matching low surrogate character.)

In scenario one, the data is most likely been imported in without proper transcoding of the data between data formats or character encodings. Possible sources of the data are ASCII-safe JSON data, where characters above U+FFFF are escaped as surrogate pairs. Other sources for of the surrogate pairs include Java data pipelines using the MUTF-8 encoding or a database using CESU-8 encoding and failure to convert between encodings when data is retrieved via SQL or exported form database.

In this discussion I will assume that U+D83D is the first character in a surrogate pair representing an emoji. This is the most common senario for the average developer, unless you are the rare breed working with historical writing systems or minority language writing systems.

For instance to get a list of emoji (using PyICU) that are represented by a surrogate pair where the high surrogate is U+D83D:

import icu
emoji = icu.UnicodeSet(r'p{Emoji}')
hs = "ud83d"  # high surrogate
hs_byte = hs.encode('utf-16-be', 'surrogatepass')
suspected_emoji = [e for e in list(emoji) if e.encode('utf-16-be').startswith(hs_byte)]
print(suspected_emoji)

Scenario one: surrogate pairs

It is necessary to first encode then decode the surrogate pair using a UTF-16 codec. A MUTF-8 or CESU-8 codec could also be used, but this aren’t standard encodings supported by Python.

emoji = 'uD83DuDC08'
print(emoji)

Will give a UnicodeEncodeError exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed

Likewise emoji.encode('utf-16').decode('utf-16') will also give result in an UnicodeEncodeError exception, so we need to use the surrogatepass error handler:

r = emoji.encode('utf-16', 'surrogatepass').decode('utf-16')
print(r)
# 🐈

When working with a dataframe:

import pandas as pd
data = {
    "user": ["Charlie", "Anna"],
    "message": ["I need to feed my uD83DuDC08", "What will you give her?"]
}

df = pd.DataFrame(data)

Various operations on the dataframe will result in will result in a UnicodeEncodeError exception. For instance df.head() will result in the error.

Like a string we can use an encode/decode sequence:

df['message'] = df['message'].str.encode('utf-16', 'surrogatepass').str.decode('utf-16')
df.head()

Scenario 2: lone surrogate

If we use the same code syntax as in scenario 1 we get a UnicodeDecodeError exception. The codec can’t handle a lone surrogate. An alternative error handler can be used, such as ignore, replace or backslashreplace. Or a custom error handler could be used:

lone = 'a lone surrogate: uD83D'
res = lone.encode('utf-16', 'surrogatepass').decode('utf-16', 'replace')
print("replace error handler: ", res)
# replace error handler:  a lone surrogate: �

Translating that to a dataframe:

import pandas as pd
data = {
    "user": ["Charlie", "Anna"],
    "message": ["I need to feed my uD83D", "What will you give her?"]
}

df = pd.DataFrame(data)
df['message'] = df['message'].str.encode('utf-16', 'surrogatepass').str.decode('utf-16', 'replace')
df.head()

Addendum

If you want to get a list of all surrogate pairs with a specific high surrogate value:

import icu
non_bmp = icu.UnicodeSet(r'[[[p{Any}]-[u0000-uFFFF]]-[p{Unassigned}]]')
hs_byte = "ud83d".encode('utf-16-be', 'surrogatepass')
suspected_chars = [e for e in list(non_bmp) if e.encode('utf-16-be').startswith(hs_byte)]

Custom error handler

It is possible to write a custom error handler that doesn’t mask the problem character:

def cp_replace_handler(exc):
  if isinstance(exc, (UnicodeEncodeError, UnicodeTranslateError)):
    s = []
    for c in exc.object[exc.start:exc.end]:
      s.append(f'<{ord(c):04X}>')
    return (''.join(s), exc.end)
  else:
    raise TypeError("can't handle %s" % exc.__name__)
codecs.register_error('cp_replace', cp_replace_handler)

lone = 'a lone surrogate: ud83d'
r1 = lone.encode('utf-16', 'cp_replace').decode('utf-16')
print(r1)
# a lone surrogate: <D83D>

surrogates = 'a surrogate pair: uD83DuDC08'
r2 = surrogates.encode('utf-16', 'cp_replace').decode('utf-16')
print(r2)
# a surrogate pair: <D83D><DC08>