I’m using a tool called httpx, using Python code. The commands I used are as follows.

import os
os.system(cat all_contents.txt | httpx -sc -title)

I want to collect the possible domains into a txt file and check if they are working properly with httpx. Besides that command, I’m running several commands together through os.system, but httpx is the only problem. Compared to when I run httpx command in a terminal window, when I run a Python file, it returns a ridiculously short result.

If httpx doesn’t work at all, I definitely think I misused the code. But in this case, the problem is that httpx works well, but the execution results don’t work for long.My guess is, “There is a difference in the output of the results between the terminal and the execution of the Python code.” When I run the Python code, it seems that something interferes with the output, or I don’t get all the results executed in the window. But I don’t know the exact reason.

What I tried was:

1) Change os.system to subprocess Popen command

import subprocess

command = "cat all_contents.txt | httpx -sc -title"
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, 
stderr=subprocess.PIPE)
stdout, stderr = process.communicate()
print("STDOUT:", stdout.decode())
print("STDERR:", stderr.decode())

This returns short results, as is the case with os.system, and when I output an error, the command returns 0 that means operating normally.

2) Change os.system to subprocess check_output command

import os

command = "cat all_contents.txt | httpx -sc -title"
result = subprocess.check_output(command, shell=True, text=True)
print(result)

It only runs httpx, but it doesn’t even return the results that came out before.

3) Adjusting options for httpx

import os
os.system(cat all_contents.txt | httpx -sc -title -t 100 -timeout 5)

I thought maybe concurrent requests were a problem, so I adjusted the thread as well. This is also useless.

4) Adding Environmental Variables

import os

os.environ['something'] = 'example_value'

In this case, I’ve tried to go this far, but I haven’t found the correct command. If this is the right way, I need your help.

The remaining guess is, should I add something like a wait() method until the command is properly executed. However, I don’t even know if it’s right and I’m struggling with my best efforts. Please give me any advices.