dog.py, the Opposite of the Usual Linux cat Command
The reverse of the usual 'cat' command in Linux.
Written by Grant Jenks
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE
LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR
OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND,
EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE
ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.
SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY
SERVICING, REPAIR OR CORRECTION.
This work is licensed under the
Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit
or send a letter to Creative Commons, 171 Second Street, Suite 300,
San Francisco, California, 94105, US.A
The idea came from here: http://1.61803398874.com/canine/
I decided to play with this after seeing the presentation on Python coroutines
Decorator exempts us from calling .next()
cr = func(*args,**kwargs)
from threading import Thread
from Queue import Queue
Decorator that manages multi-threading using a queue.
Modified to communicate queue length for backoff. This is achieved by
trying to send None (ignored) to the threaded coroutine.
messages = Queue()
item = messages.get()
if item is GeneratorExit:
item = (yield messages.qsize())
if item is None:
Coroutine for writing text to a file.
with open(name, 'w') as file:
text = (yield)
from time import sleep
def dog(file_names, max_queue_len = 2 ** 4, buffer_len = 2 ** 22):
Read from standard input and write to multiple files.
Read/write loop pseudocode:
Fill text from stdin
Make sure all threads are ready to queue
If not, exponential backoff
Queue text on all threads
if len(file_names) == 0:
raise ValueError("no file names given")
threads = [threaded(write_file(name)) for name in file_names]
text = sys.stdin.read(buffer_len)
backoff = 1
max_cnt = max(thread.send(None) for thread in threads)
if max_cnt < max_queue_len:
backoff *= 2
for thread in threads:
if len(text) != buffer_len:
for thread in threads:
if __name__ == '__main__':
cat somefile | python dog.py file1 file2...
file_names = sys.argv[1:]