Thiết kế website giá rẻ

Question

While running a piece of driver code on an AMD64 machine, it appears that there are issues related to cache consistency or out-of-order execution. Are there any experienced engineers who can help analyze the cause?

Below is the C code for the corresponding function and the disassembled code of the compiled program.

static int check_status(struct cmd_dst *out)
{
        uint32_t loops = 0;

        printf("before loop, status is 0x%xn", out->status);

        while (*(volatile uint32_t *)&out->status) {
                _mm_pause();
                loops++;
        }

        if (out->status) {
                printf("status error with 0x%xn", out->status);
                return 1;
        }

        return 0;
}

int check_cmd{}
{
...
        if (out->status && check_status(out)) {
                printf("failed to execute cmd, status error 0x%xn", out->status);
        }
...
}



0x000000000002c1b9 <+297>:   nopl   0x0(%rax)
   0x000000000002c1c0 <+304>:   mov    0x0(%rbp),%eax
   0x000000000002c1c3 <+307>:   test   %eax,%eax
   0x000000000002c1c5 <+309>:   je     0x2c240 <check_cmd+432>
   0x000000000002c1c7 <+311>:   nopw   0x0(%rax,%rax,1)
   0x000000000002c1d0 <+320>:   pause
   0x000000000002c1d2 <+322>:   mov    0x0(%rbp),%eax
   0x000000000002c1d5 <+325>:   test   %eax,%eax
   0x000000000002c1d7 <+327>:   jne    0x2c1d0 <check_cmd+320>
   0x000000000002c1d9 <+329>:   mov    0x0(%rbp),%r9d
   0x000000000002c1dd <+333>:   test   %r9d,%r9d
   0x000000000002c1e0 <+336>:   jne    0x2c240 <check_cmd+432>
   0x000000000002c1e2 <+338>:   mov    0x1c(%rbx),%esi
   0x000000000002c1e5 <+341>:   xor    %r12d,%r12d
   0x000000000002c1e8 <+344>:   test   %esi,%esi
   0x000000000002c1ea <+346>:   js     0x2c111 check_cmd+129>
   0x000000000002c1f0 <+352>:   jmpq   0x2c120 <check_cmd+144>
   0x000000000002c1f5 <+357>:   nopl   (%rax)
   0x000000000002c1f8 <+360>:   mov    0x25cde1(%rip),%rax        # 0x288fe0
   0x000000000002c1ff <+367>:   mov    $0x3b6,%r8d
   0x000000000002c205 <+373>:   lea    0x50094(%rip),%rcx        # 0x7c2a0
   0x000000000002c20c <+380>:   lea    0x4f975(%rip),%rdx        # 0x7bb88
   0x000000000002c213 <+387>:   mov    $0x1,%esi
   0x000000000002c218 <+392>:   mov    $0xffffffea,%r12d
   0x000000000002c21e <+398>:   mov    (%rax),%rdi
   0x000000000002c221 <+401>:   xor    %eax,%eax
   0x000000000002c223 <+403>:   callq  0x5250 <__fprintf_chk@plt>
   0x000000000002c228 <+408>:   add    $0x8,%rsp
   0x000000000002c22c <+412>:   mov    %r12d,%eax

out->status points to a piece of DMA memory, initially containing a random number (this random number is actually the input data for the hardware each time the command is executed, and the input data is random each time. The hardware’s output is also written to this address). At some point, the hardware will write a 0 to out->status (indicating that the command has successfully executed). This write will only happen once and will not change afterward.
The program can be considered single-threaded, with no other thread operating on this memory region.

Under expected conditions, as long as the memory is non-zero, it will remain within the loop. Once the loop ends, this memory should be 0, therefore the function should only return 0.

However, during actual execution, it is observed that the function occasionally returns 1. At this point, printing status shows that it is the same as the value printed before the loop. After returning 1, logging out->status outside the function shows that it has become 0 again. What could be the possible reason for this?

The program’s log is as follows:

before loop, status is 0x83ae
status error with 0x83ae
failed to execute cmd, status error 0x0

Adding the volatile keyword to the access of out->status in the check after the loop makes the problem disappear:

if (*(volatile uint32_t *)&out->status) {
    printf("status error with 0x%xn", out->status);
    return 1;
}

Alternatively, inserting a memory barrier before the check also resolves the issue:

__sync_synchronize();
if (out->status) {
    printf("status error with 0x%xn", out->status);
    return 1;
}

I think the only plausible explanation for this phenomenon is out-of-order execution. From the disassembled instructions, the subsequent check also reads directly from the memory address, and its order is after the read within the while loop. Therefore, if the while loop reads a 0 and the if statement reads a non-zero value, it appears as though the read instruction in the if statement is executed before the read instruction in the while loop, leading to this result.

However, upon reviewing the AMD64 manual, I have not found any sections that support this explanation. The manual’s discussion on out-of-order execution pertains to rules when there is a dependency between reads and writes, and it does not seem to describe whether two read instructions from the same memory location could be executed out of order.

Danh mục