We have this snippet to upload a file to a web api
<code>import requests
import json
api_url = "https://ourserver/ourserverfolder/api/auth/Login"
dto = { "username": "ourusename", "password": "ourpassword" }
headers = { 'Content-Type': 'application/json', 'Accept': 'application/json' }
response = requests.post(api_url, data=json.dumps(dto), headers=headers)
response_obj = response.json()
bearer = response_obj["token"]
upload_url = "https://anotherserver/anotherserverfolder/UploadFile"
filename = r"D:bigfile.zip"
filenamewithoutfolder = filename.split('\')[-1]
with open(filename, 'rb') as f:
file = (filenamewithoutfolder, f, 'application/x-zip')
upload_files = { "file": file }
upload_headers = { 'Authorization': 'Bearer ' + bearer }
upload_response = requests.post(upload_url, files=upload_files, headers=upload_headers,
verify=False)
</code>
<code>import requests
import json
api_url = "https://ourserver/ourserverfolder/api/auth/Login"
dto = { "username": "ourusename", "password": "ourpassword" }
headers = { 'Content-Type': 'application/json', 'Accept': 'application/json' }
response = requests.post(api_url, data=json.dumps(dto), headers=headers)
response_obj = response.json()
bearer = response_obj["token"]
upload_url = "https://anotherserver/anotherserverfolder/UploadFile"
filename = r"D:bigfile.zip"
filenamewithoutfolder = filename.split('\')[-1]
with open(filename, 'rb') as f:
file = (filenamewithoutfolder, f, 'application/x-zip')
upload_files = { "file": file }
upload_headers = { 'Authorization': 'Bearer ' + bearer }
upload_response = requests.post(upload_url, files=upload_files, headers=upload_headers,
verify=False)
</code>
import requests
import json
api_url = "https://ourserver/ourserverfolder/api/auth/Login"
dto = { "username": "ourusename", "password": "ourpassword" }
headers = { 'Content-Type': 'application/json', 'Accept': 'application/json' }
response = requests.post(api_url, data=json.dumps(dto), headers=headers)
response_obj = response.json()
bearer = response_obj["token"]
upload_url = "https://anotherserver/anotherserverfolder/UploadFile"
filename = r"D:bigfile.zip"
filenamewithoutfolder = filename.split('\')[-1]
with open(filename, 'rb') as f:
file = (filenamewithoutfolder, f, 'application/x-zip')
upload_files = { "file": file }
upload_headers = { 'Authorization': 'Bearer ' + bearer }
upload_response = requests.post(upload_url, files=upload_files, headers=upload_headers,
verify=False)
When we upload smaller files, it uploads them without a noticeable CPU impact.
However, when I am trying to upload a 7G file, it occupies all cpu on my computer.
Everything just stops. Like we had came back in time and was running Windows 95 or some other operating system with cooperative multitasking.
Why?
I can also notice that the program takes up memory in a linear proportion to the time it has executed.
I have some ideas
- The code might still in some way cause the file to be stored as a byte buffer.
- The multipart/form-data creation might check the whole file to come up with a suitable delimiter? Instead of just guessing?