Sending binary file through sockets (python 3, not 2.x)

I am simply trying to understand the context of

with open ('filename.png', 'rb') as f1

as I send sockets.

I can successfully copy / clone an internal file with ‘rb’ as f1, ‘wb’ as f2, but sending that over the socket protocol is driving me bonkers.

SERVER

import socket
import time
import sys
import os
# lan scope
HOST = "80.38.100.27"
PORT = 8050

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((HOST, PORT))
sock.listen(1) # how many conenctions can it receive
# listen for connecting client and sync / socket when reached out to
conn,addr = sock.accept() 
print("Connected by ", str(addr))
i = True

# send data until i = False then drop socket
while i:
    # listen for what client tells you to do
    raw_data = conn.recv(1024)
    data = repr(raw_data)[2:-1]
    print(data)
    # if 'o' is pressed on client, send the file
    if (data == 'o')
        # below should copy the file using sock.send and client has sock.recv
        with open ('test.png','rb') as f1:
            sock.send(f1.read(1024))
            
        sock.close()
    # if 'q' is pressed on client, quit
    if (data =='q'):
        i = False
conn.close()

CLIENT


import socket
import time

HOST = "80.38.100.27"
PORT = 8050
# Create socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# attempt to connect to host
sock.connect((HOST,PORT))
connected = True

while connected:
    message = input("Your Message: ")
    sock.send(bytes(message,'UTF-8'))
    if (message == 'o'):
        with open ('test1.png','wb') as remote:
            rec = sock.recv(1024)
            remote.write(rec)
    
    if (message == 'q'):
        connected = False

sock.close()

I server side crashes with errno 32

Traceback (most recent call last):
  File "sendfile.py", line 36, in <module>
    sock.send(f1.read(1024))
socket.error: [Errno 32] Broken pipe 

My first thought was “I am reading past the end of the file” but during each write, the client side test1.png filesize is still 0 meaning no data was ever received to be written to.

The python3 official documentation page is very difficult to read and the examples only use text/strings(which I already have working fine) and not binary files. The EXTENSIVE searching only has 2.7 examples without “with open” python 3.1 examples.

Thank you. I have spent probably about 20 hours trying to figure this out.

I suspect that the socket sock on server is read-only, just for the sake of accepting clients. If you want to send something, you are supposed to use the conn variable, which is linked to this particular client.

Also, while the HOST variable on client actually is the server’s address, the same variable on server delimits clients from which to listen to connections. It seems somewhat awkward that you them both to an address like this. On server, you may leave it blank.

Finally, sending binary data is your only option, there is no catch to it. Did you read the Socket Howto?

server:

import socket

HOST = "" # accept everyone
PORT = 8050

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((HOST, PORT))
sock.listen(1)
conn, addr = sock.accept() 
print("Connected by ", str(addr))

while 1:
	# listen for what client tells you to do
	raw_data = conn.recv(1024)
	data = repr(raw_data)[2:-1]
	print(data)
	# if 'o' is pressed on client, send the file
	if (data == 'o'):
		with open ('test.png','rb') as f1:
			conn.send(f1.read(1024))
	
	# if 'q' is pressed on client, quit
	elif (data =='q'):
		break
conn.close()

client:

import socket

HOST = "localhost"
PORT = 8050
# Create socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# attempt to connect to host
sock.connect((HOST,PORT))

while 1:
	message = input("Your Message: ")
	sock.send(bytes(message,'UTF-8'))
	if (message == 'o'):
		with open('test1.png', 'wb+') as output:
			rec = sock.recv(1024)
			output.write(rec)
	
	elif (message == 'q'):
		break

sock.close()

Another point is that ‘send’ and ‘recv’ are not guaranteed to process your requested 1024 bytes for sending or reading. It is possible that your ‘send’ operation will send less than 1024 bytes (you have to check the return value of the call) and also that the ‘recv’ call will not load in a full 1024 bytes either.

Thank you for the replies. I didn’t even know about docs.python.org/3/howto so sadly no.

Regarding the 1024 in and out, I am simply in the learning process of sockets and I learn via trial and error / hands on. I will keep in mind to have a call a “bytes received” and track it. Do you know if MTU size and packet size are one and the same? I was going to eventually (once I have the basic stuff down) figure out a way to do object.size() and pass it over as a varialbe for socket_name.recv(object.size()) , if it was possible.

Thank you again

I don’t know off the top of my head. In general you don’t worry about things like MTU at the application layer. The only time you really want to think about the size of the buffers you are sending is when you want to optimize your thoughput. If you have 1,000,000 bytes to send, you don’t want to make 1,000,000 calls to ‘send’ with 1 byte. Likewise, you don’t want to make one call with the 1,000,000 bytes because that will overload your NIC. The sweet spot for most OSs is to send some power of 2 value like 4096 or 8192.

now I am running into a issue of only the first (bufsize) being written and not continuing to the next part of the read cycle. I am using the example posted by emu and I have a feeling I either need to change my logic or send a “message received” from client to server (since server is sending file to client). Any input?

please correct me if I am wrong but isn’t ‘with open’ supossed to be a “while file being read until EoF read” type of loop?

Nope, no loop, it is more similar to a try/except block in its function. It just opens the file and makes sure to close it no matter what disaster hits your program. documentation.

Network communication isn’t easy, and I have little experience with it. Anyway, did you already have a look at the asynchat module?

good lord that documentation hurts my brain with over complication. Thank you for the links, i will muddle through it with my feeble muggle brain

Here is my answer should anyone else ever need it

SERVER

# written for python 3.1+ coded on Python 3.3.3
############################################################################
# the point of this py program is to take a file from the server
# and push it down the client. The server waits for a response from the client
# stating that the packet send was written correctly, if not it will retry until 
# it does and move to the next packet. The recommended packet/buffer size is 4096
# but the program is written in 1024 (see known problems) to test for more 
# frequent errors.. 
# If you want to change the packet size , change the bufsize variable on both 
# client and server. mismatched variables will not work. Anything hardcoded is
# because the data being xfered will always be small
#
# Note, the order of the data sent and data received is important the client.py
# attached is written to correspond with the proper sends/receives
# 
# Known Problems: Anything past the standard 1024 packet size fails. Debug
# the logic at your own risk, this is just a blue print to get started
#
# Scott Zupek
# [email protected]
############################################################################
import socket
import time

HOST = "" # accept all connections
PORT = 8050 # can be changed, used internally, client must match
c = 0  #used to count send cycles

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((HOST, PORT))
sock.listen(1) # how many conenctions can it receive

# listen for connecting client and sync / socket when reached out to
conn,addr = sock.accept()
print("Connected by ", str(addr)) # show when connection is made, by whom

# send data until no longer needed then drop socket
while 1:
    # listen for what client tells you to do
    raw_data = conn.recv(1024) # this is characters on the client side
    data = repr(raw_data)[2:-1] # parse the info, getting rid of the byte tags
    print(data) #print what the client typed
    # if 'o' is pressed on client, send the file
    if (data == 'o'):
        # below should copy the file using sock.send and client has sock.recv
        # set the packet buffer size to send 1024 is way to small but good for testing
        bufsize = 1024
        # if you need different filenames you'll have to write a object for that
        # this was just hard coded for testing purposed
        with open ('test.msi','rb') as f1:
            # get the entire filesize, which sets the read sector to EOF
            file_size = len(f1.read())
            # reset the read file sector to the start of the file
            f1.seek(0)
            # take filesize and write it to a temp file
            with open ('temp01',mode='w', encoding='UTF-8') as f2:
                f2.write(str(file_size))
            # pass the file size over to client in a small info chunk
            with open ('temp01', 'rb') as f3:
                conn.send(f3.read(1024))
            #send the total file size off the client
            while ((c*bufsize) < (file_size)):
                send_data = f1.read(bufsize)
                conn.send(send_data)
                c+=1
                # get bool (0 is bad | 1 is good) from client
                chunk_write_flag = int(conn.recv(1024))
                while (chunk_write_flag != 1): #while whole data was not sent..retry until successful
                    conn.send(send_data)
                    #get status from client after a retry
                    chunk_write_flag = int(conn.recv(1024))
                # used on the last chunk of the file xfer
                # if the file.read() is less than buffer size do last tasks
                if ( (file_size - (c*bufsize)) < bufsize ):
                    send_data = f1.read(bufsize)
                    conn.send(send_data)
                    f1.close()
                    break
        print ('SUCCESSFULLY TRANSFERED FILE')
        sock.close()
    # if 'q' is pressed on client, quit
    if (data =='q'):
        break
conn.close()

CLIENT

# Python 3.1+ coded on Python 3.3.3
#######################################################################################
# the point of this Python program corresponds with the sendfile.py program.
# the client connects and has the option of 'o' (send file) or 'q' (quit program)
# each chunk (bufsize var) is send from server to client.  client then checks to verify
# that all data was written on client side, if not, it requests the data again until it
# is verified cliean, then moves onto the next chunk of data
#
# known probelems:  testing a 216MByte file resulted in a hang.
#                   This was successfully tested up a 2.0MByte file
#
# Scott Zupek
# [email protected]
#######################################################################################
import socket
import time

HOST = "80.38.100.27"
PORT = 8050
# Create socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# attempt to connect to host
sock.connect((HOST,PORT))

while 1:
    message = input("Your Message: ")
    sock.send(bytes(message,'UTF-8'))
    if (message == 'o'):
        c = 0
        # before starting to write new file, get file size
        file_size = int(sock.recv(1024)) # from f3 on server side
        # print (file_size)
        # set byte buffer size
        bufsize = 1024
        # start writting at beginning and use following variable to track
        write_sectors = 0
        # this only opens the file, the while loop controls when to close
        with open ('test.msi','wb+') as output:
            # while written bytes to out is less than file_size
            while (write_sectors < file_size):
                # write the BUFSIZE while the write_sectors is less than file_size
                output.write(sock.recv(bufsize))
                c +=1
                with open('test.msi', 'rb') as verify:
                    write_check = (len(verify.read()) / c)
                    verify.seek(0) # read cycle moves seek location, reset it
                    while(write_check != bufsize):
                        # where the original write started, to send back to server
                        if (c > 1): output.seek((c-1) * bufsize)
                        if (c == 1): output.seek(0)
                        # send to server that the write write was no successfull
                        sock.send(bytes('0', 'UTF-8'))
                        output.write(sock.recv(bufsize))
                        write_check = int(len(verify.read()) /c )
                        # if last packet, smaller than bufsize 
                    sock.send(bytes('1', 'UTF-8')) #send SUCCESS back to server
                if (((file_size) - (write_check * c)) < bufsize):
                    #output.write(sock.recv(bufsize))
                    verify.close()
                    #output.close()
                    file_size = 0
                write_sectors += bufsize # successful write, move 'while' check
                # add the written sectors by the bufsize.
                # example if filesize in bytes is 4096 you need to track how much
                # was written to figure out where the EOF is
            output.write(sock.recv(bufsize)) #write the last chunk missed by while loop

    elif (message == 'q'):
        break

sock.close()

Check this simple socket programming in python

http://net-informations.com/python/net/socket.htm

The sweet spot for most OSs is to send some power of 2 value like 4096 or 8192.