fsutil.exe 8dot3name set O: 1

for set enable/disable shortname creation on volume we can use

FSCTL_QUERY_PERSISTENT_VOLUME_STATE control code

with NtFsControlFile or DeviceIoControl

ULONG SetPersistentVolumeState(PCWSTR pszVolumePath, ULONG VolumeFlags)
{
    if (HANDLE hFile = fixH(CreateFileW(pszVolumePath, FILE_WRITE_ATTRIBUTES, 
        FILE_SHARE_READ|FILE_SHARE_WRITE, 0, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, 0)))
    {
        FILE_FS_PERSISTENT_VOLUME_INFORMATION PersistVolInfo = { 
            VolumeFlags, 
            PERSISTENT_VOLUME_STATE_SHORT_NAME_CREATION_DISABLED,
            1
        };

        ULONG dwError = NOERROR;
        OVERLAPPED ov = {};
        if (!DeviceIoControl(hFile, FSCTL_SET_PERSISTENT_VOLUME_STATE, &PersistVolInfo, sizeof(PersistVolInfo), 0, 0, 0, &ov))
        {
            dwError = GetLastError();
        }
        CloseHandle(hFile);

        return dwError;
    }

    return GetLastError();
}

inline HANDLE fixH(HANDLE hFile)
{
    return hFile == INVALID_HANDLE_VALUE ? 0 : hFile;
}

if we want do this for all volumes on the system, set NtfsDisable8dot3NameCreation REG_DWORD in HKLMSystemCurrentControlSetControlFileSystemNtfsDisable8dot3NameCreation as described

if we want query file shortname, use FileAlternateNameInformation. This information class is used
to query alternate name information for a file. The alternate name for
a file is its 8.3 format name (eight characters that appear before the
“.” and three characters that appear after). A file MAY have an
alternate name to achieve compatibility with the 8.3 naming
requirements of legacy applications.<101>

A FILE_NAME_INFORMATION (section 2.1.7) data element containing an 8.3 file name (section 2.1.5.2.1) is returned by the server.
if file have no short name we got STATUS_OBJECT_NAME_NOT_FOUND

for set / delete short name, we need use FileShortNameInformation.

This information class is used to change a file’s short name. If the supplied name is of zero length, the file’s existing short name, if
any, SHOULD<144> be deleted. Otherwise, the supplied name MUST be a
valid short name as specified in section 2.1.5.2.1 and be unique among
all file names and short names in the same directory as the file being
operated on. A caller changing the file’s short name MUST have
SeRestorePrivilege, as specified in [MS-LSAD] section 3.1.1.2.1. A
FILE_NAME_INFORMATION (section 2.1.7) data element containing an 8.3
file name (section 2.1.5.2.1) is provided by the client.

NTSTATUS Strip(PCWSTR pszFileName)
{
    HANDLE hFile;
    IO_STATUS_BLOCK iosb;
    UNICODE_STRING ObjectName;
    OBJECT_ATTRIBUTES oa = { sizeof(oa), 0, &ObjectName, OBJ_CASE_INSENSITIVE };
    NTSTATUS status;
    RtlInitUnicodeString(&ObjectName, pszFileName);

    if (0 <= (status = NtOpenFile(&hFile, DELETE, &oa, &iosb, FILE_SHARE_VALID_FLAGS, FILE_OPEN_REPARSE_POINT|FILE_OPEN_FOR_BACKUP_INTENT)))
    {
        union {
            FILE_NAME_INFORMATION fni;
            UCHAR buf[offsetof(FILE_NAME_INFORMATION, FileName) + (8+1+3)*sizeof(WCHAR)];
        };
        
        if (0 <= (status = NtQueryInformationFile(hFile, &iosb, &fni, sizeof(buf), FileAlternateNameInformation)))
        {
            DbgPrint("%.*wsn", fni.FileNameLength, fni.FileName);
            fni.FileNameLength = 0;
            NtSetInformationFile(hFile, &iosb, &fni, sizeof(fni), FileShortNameInformation);
        }
        
        NtClose(hFile);
    }

    return status;
}

we of course not need query FileAlternateNameInformation before set FileShortNameInformation. i do this for demo only

note that we must enable SeRestorePrivilege ( and better SeBackupPrivilege too) before do this. and open file must be open with DELETE access.

FileShortNameInformation Change the current short file name, which is supplied in a FILE_NAME_INFORMATION structure. The file must be on an
NTFS volume, and the caller must have opened the file with the
DesiredAccess DELETE flag set in the DesiredAccess parameter.

if we want do this per folder (reqursive) – we need enumerate all files in folder and do this for every file