I am reading one or more text files, using csv.reader to concatenate the files into a single list, broken down by paragraph. When I display the list in my edit window, text highlighting in the first paragraph of each file is incorrect.

I use spacy.blank() so I can tokenize the text of a paragraph. Each paragraph is saved as a dictionary that contains the spacy doc for that paragraph (text, tokens, etc.) and a currently empty list for annotation labels.

    def merge_data_files(self, flist:list):
        self.nlp = sp.blank("en")
        for file in flist:
            with open(file, 'r', encoding='utf-8') as f:
                reader = csv.reader(f, delimiter='n')
                for row in reader:
                    self.data_df.append({'Doc': self.tokenize(row[0]), 'Labels': []})

    def tokenize(self, text):
        return self.nlp(text) 

In my edit window, I catch the left mouse button release event and manage text highlighting by the cursor. The behaviour I want is: if only a portion of a span of tokens is highlighted, highlighting will expand to the beginning of the first token and the end of the last token in the span. I get this behaviour for every paragraph except the first one of each file.

Event handling code.

    def lftBtnRel(self):
        cursor = self.caseWindow.textCursor()
        if cursor.hasSelection():
            span = self.doc.char_span(cursor.selectionStart(), 
                                      cursor.selectionEnd(), 
                                      alignment_mode="expand")
            print(span, span.start_char, span.end_char)
            cursor.setPosition(span.start_char, QTextCursor.MoveMode.MoveAnchor)
            cursor.movePosition(QTextCursor.MoveOperation.NextCharacter, 
                                QTextCursor.MoveMode.KeepAnchor, 
                                span.end_char - span.start_char)
            self.caseWindow.setTextCursor(cursor)

    def eventFilter(self, obj, event):
        if event.type() == QEvent.Type.MouseButtonRelease:
            if event.button() == Qt.MouseButton.LeftButton:
                self.lftBtnRel()

        return super().eventFilter(obj, event)

This example text demonstrates the problem. This is the first paragraph of one or my text files.

In the first part of this book, and particularly in Chapters 4–7, I have presented the basis of this book’s argument by presenting some of the salient characteristics of my approach, such as taking a long-term ex-ante perspective, learning from the past about the present for the future, using complex systems thinking, etc.

Highlighting ‘4’ in the token ‘4-7’ should highlight the entire token, but nothing gets highlighted. Investigation showed that highlighting the first character of any token results in no highlighting, because the value returned in the span is the preceding space. If any other part of a token is highlighted (say ‘e’ in the token ‘learning’) all but first character of the token is highlighted, with the addition of the following space. That is, highlighting ‘e’ in ‘learning’ results in the span of ‘earning ‘ being selected for highlighting.

Again, this only happens for the first paragraph of each files concatenated in the list, every other paragraph is handled correctly.

I’m at a loss as to whether this is a problem with csv.reader, with spacy and char_span or some combination. Any help or ideas would be greatly appreciated.