Apologies if the title is not clear—this is my first post on stack overflow. I will try to provide as much context as possible.

Basically, I have a dataset containing english statements. Some of the statements contain leading/trailing punctuation, so I would like to clean and standardize them. The criteria for standardization is that all leading/trailing punctuation is removed, the first character of the statement is capitalized, and the statement ends in a period. I have a function that does this and apply it to each row in the dataset. Here is a snippet if it helps:

def normalize_statement(statement):
    # remove leading/trailing punctuation and excess whitespace
    cleaned = statement.strip(whitespace + punctuation)
    cleaned = re.sub(r's+', ' ', cleaned)

    # capitalize first letter 
    cleaned = cleaned[0].upper() + cleaned[1:]

    # ensure the statement ends with a period
    if cleaned and not cleaned.endswith('.'):
            cleaned += '.'
    return cleaned

This function works for basic statements, but doesn’t always work with substrings wrapped between double or single quotation marks, especially if the quoted substring appears in the beginning and/or end of the statement because then one of the quotation marks gets removed after leading punctuation is removed (ie. ‘”this” is a statement’ would become ‘This” is a statement’). The intent should be to preserve the quotation marks around any substring after cleaning.

I thought about using regex to capture all substrings in between single or double quotation marks, replace all quoted substrings with the captured groups to remove the quotation marks, standardize the statement, then add back the quoted substrings in their original positions. Here is the updated function:

def normalize_statement(statement):
    # extract any and all substrings enclosed in quotation marks
    quoted_pattern = re.compile(r'["'](.*?)["']')
    quoted_substrings = quoted_pattern.findall(statement)

    # replace the quoted substrings with the captured groups
    cleaned = re.sub(quoted_pattern, lambda m: m.group(1), statement)

    # remove leading/trailing punctuation and excess whitespace
    cleaned = cleaned.strip(whitespace + punctuation)
    cleaned = re.sub(r's+', ' ', cleaned)

    # capitalize first letter for languages that use capitalization
    cleaned = cleaned[0].upper() + cleaned[1:]

    # ensure the statement ends with a period
    if cleaned and not cleaned.endswith('.'):
            cleaned += '.'

    # replace quoted substrings (if any) back into their original positions after cleaning
    for quoted_substring in quoted_substrings:
        cleaned = re.sub(re.escape(quoted_substring.strip(whitespace + punctuation)), f'"{quoted_substring}"', cleaned, flags=re.IGNORECASE)

    return cleaned

While the regex pattern ‘”‘[“‘]’ works sometimes (ie. Employees are “expected” to embrace ‘frugality’.), it fails for statements like ‘There’s no place like home, emphasizing the unique comfort of one’s own surroundings.’ because the captured group in this case would be the substring between the apostrophes: ‘s no place like home, emphasizing the unique comfort of one’. I only want text between quotations to be captured—but how can I differentiate between apostrophes and single quotes in my regex pattern? I’m not too familiar with regex, so if anyone has a different approach, that would be greatly appreciated!